Implementation¶

Forward Mode Implementation¶

Variable domain: The variables are defined as real numbers, hence any calculations or results involving complex numbers will be excluded from the package.
Type of user input: Regardless of the input type (e.g., an int, a float or a list or a numpy array), the Var class will automatically convert the input into a numpy array.
Core data structures: The core data structures will be classes, lists and numpy arrays.
- Classes will help us provide an API for differentiation and custom functions, including custom methods for our elementary functions.
- Numpy arrays are the main data structure during the calculation. We store the list of derivatives as a numpy array so that we can apply entire functions to the array, rather than to each entry separately. Each trace Var has a numpy array of derivatives where the length of the array is the number of input variables in the function. In the vector-vector case, if we have a function \(f: \mathbb{R}^m \rightarrow \mathbb{R}^n\) or \(f: \mathbb{R}^1 \rightarrow \mathbb{R}^n\), we can process this as \(f = [f_1, f_2, \ldots, f_n]\), where each \(f_i\) is a function \(f_i: \mathbb{R}^m \rightarrow \mathbb{R}\). Our implementation can act as a wrapper over these functions, and we can evaluate each \(f_i\) independently, so long as we define \(f_i\) in terms of the \(m\) inputs. Currently, the module supports both scalar to scalar, scalar to vector, vector to scalar and vector to vector functions.
Our implementation plan includes 1 class which accounts for trace variables and derivatives with respect to each input variable.
- Var class. The class instance itself has two main attributes: the value and the evaluated derivatives (Jacobian) with respect to each input. Within the class we redefine the elementary functions and basic algebraic functions, including both evaluation and derivation. Since our computation graph includes “trace” variables, this class will account for each variable.

API¶

`DeriveAlive.DeriveAlive`¶

Class attributes and methods:
- Attributes in Var: self.var, self.der. To cover vector-to-vector cases, we implement our self.var and self.der as numpy arrays, in order to account for derivatives with respect to each input variable. Also the constructor checks whether the values and derivatives are integers, floats, or lists, and transforms them into numpy arrays automatically.
- We have overloaded elementary mathematical operations such as addition, subtraction, multiplication, division, sine, pow, log, etc. that take in 1 Var type, or 2 types, or 1 Var type and 1 constant, and return a new Var (i.e. the next “trace” variable). All other operations on constants will use the standard Python library. In each Var, we will store as attributes the value of the variable (which is calculated based on the current operation and previous trace variables) and the evaluated gradient of the variable with respect to each input variable.
- Methods in Var:
  - __init__: initialize a Var class object, regardless of the user input, with values and derivatives stored as numpy arrays.
  - __repr__: overload the print format, prints self in the form of Var([val], [der]) when self is a scalar or constant; prints self in the form of Values([val]) Jacobian([der]) when self is a vector.
  - __add__: overload add function to handle addition of Var class objects and addition of Var and non-Var objects.
  - __radd__: preserve addition commutative property.
  - __sub__: overload subtraction function to handle subtraction of Var class objects and subtraction between Var and non-Var objects.
  - __rsub__: allow subtraction for \(a - \texttt{Var}\) case where a is a float or an integer.
  - __mul__: overload multiplication function to handle multiplication of Var class objects and multiplication between Var and non-Var objects.
  - __rmul__: preserve multiplication commutative property.
  - __truediv__: overload division function to handle division of Var class objects over floats or integers.
  - __rtruediv__: allow division for \(a \div \texttt{Var}\) case where \(a\) is a float or an integer.
  - __neg__: return negated Var.
  - __abs__: return the absolute value of Var.
  - __eq__: return True if two Var objects have the same value and derivative, False otherwise.
  - __ne__: return False if two Var objects have the same value and derivative, True otherwise.
  - __lt__: return True if the value of Var object is less than an integer / a float / the value of Var object, False otherwise.
  - __le__: return True if the value of Var object is less than or equal to an integer / a float / the value of Var object, False otherwise.
  - __gt__: return True if the value of Var object is greater than an integer / a float / the value of Var object, False otherwise.
  - __ge__: return True if the value of Var object is greater than or equal to an integer / a float / the value of Var object, False otherwise.
  - __pow__, __rpow__, pow: extend power functions to Var class objects.
  - sin, cos, tan: extend trigonometric functions to Var class objects.
  - arcsin, arccos, arctan: extend inverse trigonometric functions to Var class objects.
  - sinh, cosh, tanh: extend hyperbolic functions to Var class objects.
  - sqrt: return the square root of Var class objects.
  - log: extend logarithmic functions with custom base input to Var class objects.
  - exp: extend exponential functions to Var class objects.
  - logistic: return the logistic function value with input of Var objects.
External dependencies:
- NumPy - This provides an API for a large collection of high-level mathematical operations. In addition, it provides support for large, multi-dimensional arrays and matrices.
- doctest - This module searches for pieces of text that look like interactive Python sessions (typically within the documentation of a function), and then executes those sessions to verify that they work exactly as shown.
- pytest - This is an alternative, more Pythonic way of writing tests, making it easy to write small tests, yet scales to support complex functional testing. We plan to use this for a comprehensive test suite.
- setuptools - This package allows us to create a package out of our project for easy distribution. See more information on packaging instructions here: https://packaging.python.org/tutorials/packaging-projects/.
- Test suites: Travis CI, Coveralls
Elementary functions
- Our explanation of our elementary functions is included in the “Class attributes and methods” section above. For the elementary functions, we defined our own custom methods within the Var class so that we can calculate, for example, the \(\sin(x)\) of a variable \(x\) using a package such as numpy, and also store the proper gradient (\(\cos(x)dx\)) to propagate the gradients forward. For example, consider a scalar function where self.val contains the current evaluation trace and self.der is a numpy array of the derivative of the current trace with respect to the input. When we apply \(\sin\), we propagate as follows:
```
def sin(self):
           val = np.sin(self.val)
           der = np.cos(self.val) * self.der
           return Var(val, der)
```
  The structure of each elementary function is that it calculates the new value (based on the operation) and the new derivative, and then returns a new Var with the updated arguments.

`DeriveAlive.rootfinding`¶

Detailed methods with inputs and return information are listed in Additional Features - Root Finding.

Methods:
- NewtonRoot: return a root of a function \(f: \mathbb{R}^m \Rightarrow \mathbb{R}^1\)
- plot_results: See docstring.
External dependencies:
- DeriveAlive
- NumPy
- matplotlib.pyplot
- Test suites: Travis CI, Coveralls

`DeriveAlive.optimize`¶

Detailed methods with inputs and return information are listed in Additional Features - Optimization.

Methods:
- GradientDescent: solve for a local minimum of a function \(f: \mathbb{R}^m \Rightarrow \mathbb{R}^1\). If \(f\) is a convex function, then the local minimum is a global minimum.

Note

Supports data set compatibility and mean squared error optimization.

BFGS: sovle for a local stationary point, i.e. \(\nabla f =0\), of a function \(f: \mathbb{R}^m \Rightarrow \mathbb{R}^1\).
plot_results: See docstring.

External dependencies:
- DeriveAlive
- NumPy
- matplotlib.pyplot
- Test suites: Travis CI, Coveralls

`DeriveAlive.spline`¶

Detailed methods with inputs and return information are listed in Additional Features - Quadratic Splines.

Methods:
- quad_spline_coeff: calculate the coefficients of quadratic splines.
- spline_points: get the coordinates of points on the corresponding splines.
- quad_spline_plot: plot the original function and the corresponding splines.
- spline_error: calculate the average absolute error of the spline and the original function at one point.
External dependencies:
- DeriveAlive
- NumPy
- matplotlib.pyplot
- Test suites: Travis CI, Coveralls

Implementation¶

Forward Mode Implementation¶

API¶

DeriveAlive.DeriveAlive¶

DeriveAlive.rootfinding¶

DeriveAlive.optimize¶

DeriveAlive.spline¶

`DeriveAlive.DeriveAlive`¶

`DeriveAlive.rootfinding`¶

`DeriveAlive.optimize`¶

`DeriveAlive.spline`¶