I’ve spent about a year ramping up on reinforcement learning (RL) and related research topics. Compared to my PhD work at Berkeley, my current research is less focused on publishing and more independent, creating unique challenges and rewards. I want to share some tips for effective self-study and independent research, which are roughly ordered from those targeting early research first to more mature research later.

  • Write down your goals. Do you want to publish a paper? Contribute to an open source project? Gain experience for a career change? Do you simply enjoy learning? You can list more than one goal, but sort them and pick one as your primary. When I started, my primary goal was to learn about as many approaches to agent based programs as I could find, and a secondary goal was to build an agent that uses mind-wandering (which led me to build a chat bot).
  • Mix learning and practicing. I learn by reading, watching videos, taking online classes, and participating in study groups. I practice by coding. A 1:1 ratio works well for me. I usually spend days or weeks in one mode before switching to the other.
  • Regularly read classic papers. Get to know the history of the field and the key figures. Often what they’ve done is worth trying yourself, or will influence your direction.
  • Start or join a reading group. This helps with motivation and learning. Don’t overdo it: start with one paper or textbook chapter per month, adjust later if necessary. Take notes each time and consider posting them online.
  • Write down your research plan. Don’t overthink it, this is just a starting spot. It doesn’t have to be long. I keep mine in a google doc. I re-read and update it frequently. It starts high-level and aspirational and gets more detailed. Here are some sections my plan contains (yours can be much shorter!):
    • One sentence research hypothesis.
    • Objectives - one line per each of my 3mo / 6mo / 1yr / 2yr objectives. I review and revise this section every three months.
    • Draft abstract + intro of a hypothetical research paper (dissertation?) that I hope to some day write if my research works out.
    • Learning plan / courses to take.
    • Reading list & related work.
    • FAQ. Brainstorm questions that you expect to encounter, both from others and yourself. Since I shifted from lots of structure in my previous role to less structure in my current role, my FAQ includes questions like “why am i doing this?” and “what possible obstacles do I expect?”
  • Revisit and evolve your goals. My high-level goals evolve slowly, but my lower-level ones change a lot. I enjoy looking back and seeing how they’ve changed as I learned new things.
  • Conduct self-performance reviews. Independent work has little intrinsic structure. I force structure on myself: every three months I write down where I can improve the most, and what I’m most proud of. In fact, this blog post is part of my 1yr performance review.
  • Keep a research journal. Try to write daily. I keep my journal at the end of my research plan, and this blog is an experiment in shifting some of my journaling to be public.
  • Don’t worry about novelty. It is valuable to implement well known algorithms yourself. Remember to cite the author if you publish your code or anything derivitive.
  • Build small projects. Apply what you’re learning. Spend hours (not days) making a 1-2 page design doc including a 1 sentence summary of the project, 1-3 paragraphs of overview, and a bullet list of the initial features. Spend weeks (not months) building the first prototype of a project, then iterate. Some friends and I spent a couple months implementing 2048 and a bunch of algos to play it; it was motivating and we learned a lot. I also built a chat bot, a small python library of utility functions for writing RL algorithms, and a few other small projects.
  • Write up your progress. When you’ve finished a project or reached a checkpoint, take a few hours to write up your results and some reflections. It doesn’t have to be polished. My friends and I documented some of our results and lessons learned working on 2048 in a shared google doc. Now we have the option to come back later and iterate on it, maybe even work towards publishing it somewhere.
  • Teach what you learn. Prepare a short talk, or even better a short hands-on training tutorial. Give it at a meetup or put it on YouTube. If you’re not ready for that, give it for your friends.
  • Allow yourself to explore tangents. But time box them, and if possible, try to synthesize lessons from the tangent back into your research plan.
  • Find folks to collaborate with.
    • Look where you work. For academics, this is straightforward, e.g., PhD students have their advisors and other students. If you’re not an academic, there might already be reading groups or chat channels at your company you can join. Otherwise, attend meetups, post on mailing lists of open source projects, or go to a conference.
    • Once you find a collaborator, some of your solo activites can become group activities. I work closely with a friend Nick Jalbert that I met in grad school. We have weekly video chats and occasionally in-person mini-retreats. Here are examples of ways Nick and I collaborate:
      • Every few weeks, we read a paper and then meet to discuss it. Nick inspired me to set up this blog when he started posting his reading notes online.
      • We built a short hands-on Intro to RL tutorial for the LA Deep RL meetup group.
      • We are self-studying our way through Gilbert Strang’s online linear algebra course.
      • We write code together. Sometimes we both work on our own separate implementation of an algorithm and then compare, sometimes we review each other’s code or design docs, and sometimes we pair program.
  • Contact people in the field. Be bold but humble. Don’t start with the most famous researchers; instead contact folks that are local to your area. Read their recent papers or books first. Get an introduction if you can. Raise a technical topic or question that overlaps with their research. Researchers are generally meritocratic, so they will give you time if you have ideas that are interesting to them. Nonetheless, expect to be rejected or ignored.
  • Think long term. I once read that people overestimate how much progress they can make in the short term, and underestimate how much they can make in the long term; this is my experience with self-study and independent research. Major research accomplishments tend to take years of work, so think long term.
  • Spend time motivating yourself. Along your way, you should expect to ocassionally feel bored, distracted, disillusioned, etc. Be patient and work through the feelings. They will pass. To help with this, I proactively set aside time for activities that inspire me. For me, this includes reading or watching sci-fi, public speaking, and passionate conversations with friends about RL and AI. And when a bad day comes along, I take a break from work, listen to music or go the gym, talk it out with a friend, or try jumping into a different project. All the while, I remain patient for my sense of excitement to return. And it always does.

Have fun and good luck with your research and studying.