Blog

Week 1: I became familiar with higher order learning, and began thinking of an interesting project for the summer. In particular, I was looking for something that incorporated my research from last summer, which involved reinforcement learning. I read two papers and wrote summaries for them both.

Week 2: I worked out paper and pencil exercises for the math involved in the two papers I read. Then I came up with an exciting idea that will enable me to combine reinforcement learning with higher order learning! I think this idea has a lot of potential to lead to some very interesting research. I am pretty excited about it.

Week 3: This week I have been working on a simulation of my new algorithm. I started with a typical Reinforcement Learning problem, and then modified it using higher order methods. I think I have made a breakthrough-- this is converging much faster than typical RL methods!! The only problem is, it runs slower as well. Some modifications need to me made to my algorithm to make it more efficient. But this is minor. Before, it was taking hours to run, now it is only taking minutes.

Week 4: I got the program to run faster by pretty much rewriting a lot of things. There are random spikes in my episodes, so I've been looking for the cause of them.

Week 5: I've been testing the algorithm in different domains. I am pretty sure I have figured out what is causing the spikes during the convergence tests, but I have not figured out the best way to fix the problem. The algorithm is working well in a simple discrete gridworld domain (except for the spikes). I am working on getting it to work in a continuous domain, but there are a few things that need to be changed in order for this to work properly.