Vector Derivatives of a Log Expression

Formulas to address a question posted in https://mathematica.stackexchange.com/questions/262063/third-fourth-derivative-of-cross-entropy-loss/262193

Setup


Results

• Original expression:

In[]:=

xent

Out[]=

Log

n

∑

i

z

i



-

n

∑

i

q

i

z

i

• First derivative:

In[]:=

D[xent,z[i]]

Out[]=

n

∑

K[1]

z

K[1]



δ

i,K[1]

n

∑

i

z

i



-

n

∑

K[1]

δ

i,K[1]

q

K[1]

In[]:=

%//DeltaSimplify

Out[]=

-

q

i

+

z

i



n

∑

i

z

i



In[]:=

%//pSimplify

Out[]=

p

i

-

q

i

• Second derivative:

In[]:=

D[xent,z[i],z[j]]

Out[]=

-

n

∑

K[1]

0-



n

∑

i

z

i



δ

i,j



n

∑

K[1]

z

K[1]



δ

i,K[1]

2



n

∑

i

z

i





+

n

∑

K[1]

z

K[1]



δ

i,K[1]

δ

j,K[1]

n

∑

i

z

i



In[]:=

%//DeltaSimplify

Out[]=

-

z

i

+

z

j



2



n

∑

i

z

i





+

z

i



δ

i,j

n

∑

i

z

i



In[]:=

%//pSimplify

Out[]=

δ

i,j

p

i

-

p

i

p

j

• Third derivative:

In[]:=

D[xent,z[i],z[j],z[k]]

Out[]=

-

n

∑

K[1]

0+

2

n

∑

i

z

i



δ

i,j



n

∑

i

z

i



δ

i,k



n

∑

K[1]

z

K[1]



δ

i,K[1]

3



n

∑

i

z

i





-



n

∑

i

z

i



δ

i,j

δ

i,k



n

∑

K[1]

z

K[1]



δ

i,K[1]

2



n

∑

i

z

i





-



n

∑

i

z

i



δ

i,k



n

∑

K[1]

z

K[1]



δ

i,K[1]

δ

j,K[1]

2



n

∑

i

z

i





-



n

∑

i

z

i



δ

i,j



n

∑

K[1]

z

K[1]



δ

i,K[1]

δ

k,K[1]

2



n

∑

i

z

i





+

n

∑

K[1]

z

K[1]



δ

i,K[1]

δ

j,K[1]

δ

k,K[1]

n

∑

i

z

i



In[]:=

%//DeltaSimplify

Out[]=

2

z

i

+

z

j

+

z

k



3



n

∑

i

z

i





-

z

i

+

z

k



δ

i,j

2



n

∑

i

z

i





-

z

i

+

z

j



δ

i,k

2



n

∑

i

z

i





-

z

i

+

z

j



δ

j,k

2



n

∑

i

z

i





+

z

i



δ

i,j

δ

i,k

n

∑

i

z

i



In[]:=

%//pSimplify

Out[]=

δ

i,j

δ

i,k

p

i

-

δ

i,k

p

i

p

j

-

δ

j,k

p

i

p

j

-

δ

i,j

p

i

p

k

+2

p

i

p

j

p

k

Obviously the result is highly symmetric (as it must be due to the symmetry of partial derivatives).

• Fourth derivative:

In[]:=

D[xent,z[i],z[j],z[k],z[l]]//DeltaSimplify//pSimplify

Out[]=

δ

i,j

δ

i,k

δ

i,l

p

i

-

δ

i,k

δ

i,l

p

i

p

j

-

δ

i,l

δ

j,k

p

i

p

j

-

δ

i,k

δ

j,l

p

i

p

j

-

δ

j,k

δ

j,l

p

i

p

j

-

δ

i,j

δ

i,l

p

i

p

k

-

δ

i,j

δ

k,l

p

i

p

k

+2

δ

i,l

p

i

p

j

p

k

+2

δ

j,l

p

i

p

j

p

k

+2

δ

k,l

p

i

p

j

p

k

-

δ

i,j

δ

i,k

p

i

p

l

+2

δ

i,k

p

i

p

j

p

l

+2

δ

j,k

p

i

p

j

p

l

+2

δ

i,j

p

i

p

k

p

l

-6

p

i

p

j

p

k

p

l

• Fifth derivative:

In[]:=

D[xent,z[i],z[j],z[k],z[l],z[m]]//DeltaSimplify//pSimplify

Out[]=

δ

i,j

δ

i,k

δ

i,l

δ

i,m

p

i

-

δ

i,k

δ

i,l

δ

i,m

p

i

p

j

-

δ

i,l

δ

i,m

δ

j,k

p

i

p

j

-

δ

i,k

δ

i,m

δ

j,l

p

i

p

j

-

δ

i,m

δ

j,k

δ

j,l

p

i

p

j

-

δ

i,k

δ

i,l

δ

j,m

p

i

p

j

-

δ

i,l

δ

j,k

δ

j,m

p

i

p

j

-

δ

i,k

δ

j,l

δ

j,m

p

i

p

j

-

δ

j,k

δ

j,l

δ

j,m

p

i

p

j

-

δ

i,j

δ

i,l

δ

i,m

p

i

p

k

-

δ

i,j

δ

i,m

δ

k,l

p

i

p

k

-

δ

i,j

δ

i,l

δ

k,m

p

i

p

k

-

δ

i,j

δ

k,l

δ

k,m

p

i

p

k

+2

δ

i,l

δ

i,m

p

i

p

j

p

k

+2

δ

i,m

δ

j,l

p

i

p

j

p

k

+2

δ

i,l

δ

j,m

p

i

p

j

p

k

+2

δ

j,l

δ

j,m

p

i

p

j

p

k

+2

δ

i,m

δ

k,l

p

i

p

j

p

k

+2

δ

j,m

δ

k,l

p

i

p

j

p

k

+2

δ

i,l

δ

k,m

p

i

p

j

p

k

+2

δ

j,l

δ

k,m

p

i

p

j

p

k

+2

δ

k,l

δ

k,m

p

i

p

j

p

k

-

δ

i,j

δ

i,k

δ

i,m

p

i

p

l

-

δ

i,j

δ

i,k

δ

l,m

p

i

p

l

+2

δ

i,k

δ

i,m

p

i

p

j

p

l

+2

δ

i,m

δ

j,k

p

i

p

j

p

l

+2

δ

i,k

δ

j,m

p

i

p

j

p

l

+2

δ

j,k

δ

j,m

p

i

p

j

p

l

+2

δ

i,k

δ

l,m

p

i

p

j

p

l

+2

δ

j,k

δ

l,m

p

i

p

j

p

l

+2

δ

i,j

δ

i,m

p

i

p

k

p

l

+2

δ

i,j

δ

k,m

p

i

p

k

p

l

+2

δ

i,j

δ

l,m

p

i

p

k

p

l

-6

δ

i,m

p

i

p

j

p

k

p

l

-6

δ

j,m

p

i

p

j

p

k

p

l

-6

δ

k,m

p

i

p

j

p

k

p

l

-6

δ

l,m

p

i

p

j

p

k

p

l

-

δ

i,j

δ

i,k

δ

i,l

p

i

p

m

+2

δ

i,k

δ

i,l

p

i

p

j

p

m

+2

δ

i,l

δ

j,k

p

i

p

j

p

m

+2

δ

i,k

δ

j,l

p

i

p

j

p

m

+2

δ

j,k

δ

j,l

p

i

p

j

p

m

+2

δ

i,j

δ

i,l

p

i

p

k

p

m

+2

δ

i,j

δ

k,l

p

i

p

k

p

m

-6

δ

i,l

p

i

p

j

p

k

p

m

-6

δ

j,l

p

i

p

j

p

k

p

m

-6

δ

k,l

p

i

p

j

p

k

p

m

+2

δ

i,j

δ

i,k

p

i

p

l

p

m

-6

δ

i,k

p

i

p

j

p

l

p

m

-6

δ

j,k

p

i

p

j

p

l

p

m

-6

δ

i,j

p

i

p

k

p

l

p

m

+24

p

i

p

j

p

k

p

l

p

m

• Sixth derivative:

In[]:=

D[xent,z[i],z[j],z[k],z[l],z[m],z[n]]//DeltaSimplify//pSimplify

Out[]=