Vector Derivatives of a Log Expression
Vector Derivatives of a Log Expression
Formulas to address a question posted in https://mathematica.stackexchange.com/questions/262063/third-fourth-derivative-of-cross-entropy-loss/262193
Setup
Setup
Results
Results
• Original expression:
In[]:=
xent
Out[]=
Log-
n
∑
i
z
i
n
∑
i
q
i
z
i
• First derivative:
In[]:=
D[xent,z[i]]
Out[]=
n
∑
K[1]
z
K[1]
δ
i,K[1]
n
∑
i
z
i
n
∑
K[1]
δ
i,K[1]
q
K[1]
In[]:=
%//DeltaSimplify
Out[]=
-+
q
i
z
i
n
∑
i
z
i
In[]:=
%//pSimplify
Out[]=
p
i
q
i
• Second derivative:
In[]:=
D[xent,z[i],z[j]]
Out[]=
-0-+
n
∑
K[1]
n
∑
i
z
i
δ
i,j
n
∑
K[1]
z
K[1]
δ
i,K[1]
2
n
∑
i
z
i
n
∑
K[1]
z
K[1]
δ
i,K[1]
δ
j,K[1]
n
∑
i
z
i
In[]:=
%//DeltaSimplify
Out[]=
-++
z
i
z
j
2
n
∑
i
z
i
z
i
δ
i,j
n
∑
i
z
i
In[]:=
%//pSimplify
Out[]=
δ
i,j
p
i
p
i
p
j
• Third derivative:
In[]:=
D[xent,z[i],z[j],z[k]]
Out[]=
-0+---+
n
∑
K[1]
2
n
∑
i
z
i
δ
i,j
n
∑
i
z
i
δ
i,k
n
∑
K[1]
z
K[1]
δ
i,K[1]
3
n
∑
i
z
i
n
∑
i
z
i
δ
i,j
δ
i,k
n
∑
K[1]
z
K[1]
δ
i,K[1]
2
n
∑
i
z
i
n
∑
i
z
i
δ
i,k
n
∑
K[1]
z
K[1]
δ
i,K[1]
δ
j,K[1]
2
n
∑
i
z
i
n
∑
i
z
i
δ
i,j
n
∑
K[1]
z
K[1]
δ
i,K[1]
δ
k,K[1]
2
n
∑
i
z
i
n
∑
K[1]
z
K[1]
δ
i,K[1]
δ
j,K[1]
δ
k,K[1]
n
∑
i
z
i
In[]:=
%//DeltaSimplify
Out[]=
2++
z
i
z
j
z
k
3
n
∑
i
z
i
z
i
z
k
δ
i,j
2
n
∑
i
z
i
z
i
z
j
δ
i,k
2
n
∑
i
z
i
z
i
z
j
δ
j,k
2
n
∑
i
z
i
z
i
δ
i,j
δ
i,k
n
∑
i
z
i
In[]:=
%//pSimplify
Out[]=
δ
i,j
δ
i,k
p
i
δ
i,k
p
i
p
j
δ
j,k
p
i
p
j
δ
i,j
p
i
p
k
p
i
p
j
p
k
Obviously the result is highly symmetric (as it must be due to the symmetry of partial derivatives).
• Fourth derivative:
In[]:=
D[xent,z[i],z[j],z[k],z[l]]//DeltaSimplify//pSimplify
Out[]=
δ
i,j
δ
i,k
δ
i,l
p
i
δ
i,k
δ
i,l
p
i
p
j
δ
i,l
δ
j,k
p
i
p
j
δ
i,k
δ
j,l
p
i
p
j
δ
j,k
δ
j,l
p
i
p
j
δ
i,j
δ
i,l
p
i
p
k
δ
i,j
δ
k,l
p
i
p
k
δ
i,l
p
i
p
j
p
k
δ
j,l
p
i
p
j
p
k
δ
k,l
p
i
p
j
p
k
δ
i,j
δ
i,k
p
i
p
l
δ
i,k
p
i
p
j
p
l
δ
j,k
p
i
p
j
p
l
δ
i,j
p
i
p
k
p
l
p
i
p
j
p
k
p
l
• Fifth derivative:
In[]:=
D[xent,z[i],z[j],z[k],z[l],z[m]]//DeltaSimplify//pSimplify
Out[]=
δ
i,j
δ
i,k
δ
i,l
δ
i,m
p
i
δ
i,k
δ
i,l
δ
i,m
p
i
p
j
δ
i,l
δ
i,m
δ
j,k
p
i
p
j
δ
i,k
δ
i,m
δ
j,l
p
i
p
j
δ
i,m
δ
j,k
δ
j,l
p
i
p
j
δ
i,k
δ
i,l
δ
j,m
p
i
p
j
δ
i,l
δ
j,k
δ
j,m
p
i
p
j
δ
i,k
δ
j,l
δ
j,m
p
i
p
j
δ
j,k
δ
j,l
δ
j,m
p
i
p
j
δ
i,j
δ
i,l
δ
i,m
p
i
p
k
δ
i,j
δ
i,m
δ
k,l
p
i
p
k
δ
i,j
δ
i,l
δ
k,m
p
i
p
k
δ
i,j
δ
k,l
δ
k,m
p
i
p
k
δ
i,l
δ
i,m
p
i
p
j
p
k
δ
i,m
δ
j,l
p
i
p
j
p
k
δ
i,l
δ
j,m
p
i
p
j
p
k
δ
j,l
δ
j,m
p
i
p
j
p
k
δ
i,m
δ
k,l
p
i
p
j
p
k
δ
j,m
δ
k,l
p
i
p
j
p
k
δ
i,l
δ
k,m
p
i
p
j
p
k
δ
j,l
δ
k,m
p
i
p
j
p
k
δ
k,l
δ
k,m
p
i
p
j
p
k
δ
i,j
δ
i,k
δ
i,m
p
i
p
l
δ
i,j
δ
i,k
δ
l,m
p
i
p
l
δ
i,k
δ
i,m
p
i
p
j
p
l
δ
i,m
δ
j,k
p
i
p
j
p
l
δ
i,k
δ
j,m
p
i
p
j
p
l
δ
j,k
δ
j,m
p
i
p
j
p
l
δ
i,k
δ
l,m
p
i
p
j
p
l
δ
j,k
δ
l,m
p
i
p
j
p
l
δ
i,j
δ
i,m
p
i
p
k
p
l
δ
i,j
δ
k,m
p
i
p
k
p
l
δ
i,j
δ
l,m
p
i
p
k
p
l
δ
i,m
p
i
p
j
p
k
p
l
δ
j,m
p
i
p
j
p
k
p
l
δ
k,m
p
i
p
j
p
k
p
l
δ
l,m
p
i
p
j
p
k
p
l
δ
i,j
δ
i,k
δ
i,l
p
i
p
m
δ
i,k
δ
i,l
p
i
p
j
p
m
δ
i,l
δ
j,k
p
i
p
j
p
m
δ
i,k
δ
j,l
p
i
p
j
p
m
δ
j,k
δ
j,l
p
i
p
j
p
m
δ
i,j
δ
i,l
p
i
p
k
p
m
δ
i,j
δ
k,l
p
i
p
k
p
m
δ
i,l
p
i
p
j
p
k
p
m
δ
j,l
p
i
p
j
p
k
p
m
δ
k,l
p
i
p
j
p
k
p
m
δ
i,j
δ
i,k
p
i
p
l
p
m
δ
i,k
p
i
p
j
p
l
p
m
δ
j,k
p
i
p
j
p
l
p
m
δ
i,j
p
i
p
k
p
l
p
m
p
i
p
j
p
k
p
l
p
m
• Sixth derivative:
In[]:=
D[xent,z[i],z[j],z[k],z[l],z[m],z[n]]//DeltaSimplify//pSimplify
Out[]=