3D Printable Device for Automated Operant Conditioning in the Mouse

Visual Abstract


Introduction
Operant conditioning (OC; Jones et al., 1939) is a standard technique used in experimental psychology in which animals, like rodents (Francis and Kanold, 2017;O'Leary et al., 2018), reptiles (Mueller-Paul et al., 2014), birds (Cook, 1992), dogs (Range et al., 2008), monkeys (Range et al., 2008), and humans (Siqueland, 1964;Angulo-Barroso et al., 2017), learn to perform an action to achieve a reward. By using this paradigm, it is possible to extract learning curves and measure accurately mental chronometry [e.g., reaction times (RTs)]. As previously suggested (Escobar and Pérez-Herrera, 2015;Francis and Kanold, 2017;O'Leary et al., 2018), different products for OC are commercially available, though most of them do not provide customizable features and are relatively expensive. Neuroscience research has greatly benefited from new 3D printing technologies bringing new possibilities to build tools, and increasing productivity and user timeliness. 3D printing also opened unprecedented resources for training students and solving common experimental problems (Baden et al., 2015). There is a plethora of work using 3Dprinted mechanical parts (Baden et al., 2015), ranging from fluorescence microscopes (Chagas et al., 2017) to electrophysiology systems (Siegle et al., 2017). The combination of 3D printing with off-the-shelf, low-cost optical and electronic components facilitates reproducibility of experimental tools internationally and promotes rapid iteration and prototyping (Chagas et al., 2017). Here, we demonstrate an affordable, fully 3D printable, and automated solution that can be reproduced rigorously in any laboratory equipped with a 3D printer with a total cost around 160 e (Table 1). We designed the chamber entirely using 3D modeling for several reasons: first, it has a high degree of reproducibility, since the model is standardized and can be downloaded to print the same structure with the same materials throughout different laboratories. Second, it can be easily customized in relation with specific experimental needs. Lastly, it can be easily shared through on-line repositories. With these cost-efficient and accessible components, we assayed the possibility to perform two-alternative forced choice OC using audiovisual cues while tracking in real time mouse position.

Mice housing and handling
Animals were kept at a constant temperature (22°C) with a standard 12/12 h light/dark cycle (7 A.M. to 7 P.M.). Food was available ad libitum and changed weekly. During OC protocol, mice are water restricted (body weight .85%; Goltstein et al., 2018) of their baseline. Before the experiment mice are handled for 1 h/d for one week. After the last daily session, mice had free access to water for 1 h (23 h of water deprivation). All the experiments were conducted in accordance with the directives of European Community Council (2011/63/EU) and approved by the Italian Ministry of Health. We tested six wild-type C57BL/6J [from postnatal day (P)50 to P180, four female and two male mice; Charles River].

3D-printed OC chamber
The OC arena (16 Â 16 Â 16 cm; thickness, 3 mm; Fig.  1A) is 3D printed using gray or white PLA (B06W568X1G, Technology Outlet). The 3D project is designed using FreeCAD software, exported in stereolithography (STL) format, converted to G-code using Cura (https://ultimaker. com/software/ultimaker-cura) and printed using Kentstrapper Verve 3D printer (https://kentstrapper.com/stampante-3d-kentstrapper-verve/). In Figure 1C, an exploded-view drawing of the chamber is shown. The color coding corresponds to different components of the apparatus (visual stimulation parts in red; camera holder in green; syringe pump in purple). All these components are coated using epoxy transparent resin (LF-L2GR-26GX, resinpro), which allows cleaning (5% ethanol in water). The arena front wall contains the elements interfacing the animal with the computer. It can be modularly assembled to the arena and is composed by a squared frame containing the LED matrix at the center, four holes for the touch buttons in the lower part, a central hole for the lick spout and a hole in the upper part to connect a piezo buzzer. The touch buttons are printed using graphene PLA (PLA_GRAFENE_175, filoprint), and connected using conductive glue (Chemtronics: CW2400) to a female pin (B07XQHD752, amazon.it) using a resistor (25 MV). A dotted grid is interposed between the LED matrix and the inside of the chamber and has two roles: first, the dotted pattern restricts the visibility of the LED lights to equal small circles; second, it contains a grid of walls facing the LEDs that prevents the light from each source to spill over to the neighboring dots. The LED matrix is covered with a thin white Plexiglas foil, so that single LED are not visible if they are off and to diffuse light uniformly. The camera holder, is joint assembled on top of the frontal wall and it is designed to maintain the camera at the distance necessary to image the entire arena using a 3.6mm focal length objective. The syringe pump is composed of a base that fixes the barrel of the syringe into position and of a piston that slides on a stepper motor guided M8 metal screw and allows to push or pull the plunger.

Hardware
An electronic board is mounted on a grounded metal sheet and is composed by a Raspberry Pi connected via USB to an Arduino UNO (AU) board (https://store. arduino.cc/arduino-uno-rev3). The Raspberry Pi (https:// www.raspberrypi.org/products/raspberry-pi-3-model-b/) acts as the main computer of the setup. It executes the Python three script that handles the structure of the experiment, performs computer vision using a Raspberry PI camera (Bewinner: Bewinnertyv48w6mf5), and saves data (Fig. 1B). The AU controls sensors and actuators in  Open Source Tools and Methods the OC chamber. Two touch buttons, made using conductive PLA, acts as capacitive sensors and are connected to AU using coaxial cables (3 mm in diameter) to minimize environmental noise. The main advantage of using graphene PLA resides in the possibility to print different button designs (e.g., for motor impairment, nose poking, etc.). There are three actuators: a LED matrix serves as display (Adafruit: 1487), a piezo buzzer (Adafruit: PS1240, frequency range: 2-10 kHz, 60 dB) is used as acoustic stimulator glued at the top of the frontal door, and a stepper motor (amazon.it: 28BYJ-48, with ULN2003) connected to a M8 screw guiding the piston of a syringe pump controlling a disposable syringe (10 ml) connected with a silicone tube equipped by luer tapers adapters to a blunt needle (Warner Instruments: SN-18) for reward delivery. This modular configuration allows the proper cleaning of the delivery tubing after each session. We use an external 5-V 2-A DC power supply (Samsung: TA10EWE) with a 1000-mF capacitor to power the LED matrix and the stepper motor. A diagram of the electrical wiring is shown in the Figure 1D.

AU program
The code controlling the OC box is organized in four files, the file called skinner.ino contains the logic of the experiment and manages the serial communication with the computer. Different files are dedicated to different aspects of the program: the file called button.ino contains functions to control the touch buttons and play auditory stimuli, the file called ledLib.ino contains wrapper functions to control Adafruit NeoPixel library (https:// www.adafruit.com/product/1487) and generate simple visual stimuli easily, the third file called stepper.ino, contains functions to control the syringe pump using the Arduino Stepper motor library (https://www.arduino.cc/ en/reference/stepper). In summary, to setup the AU, a user needs to download the folder containing the .ino files, uncompress and upload the file skinner.ino.

Raspberry Pi program
On the Raspberry Pi, a Python script controlling the experiment has been written using IDLE. The program relies on a number of external libraries that are required to run all Open Source Tools and Methods parts of the script with no errors. Since the task relies on real time tracking of the animal position we use picamera and opencv libraries to acquire frames and process them using K-nearest neighbors based background-foreground segmentation (Zivkovic and van der Heijden, 2006), a widely used algorithm for generating a foreground mask using static cameras ( Fig. 2A). The technique consists of two main steps, the first one is the background initialization in which we use 1000 frames of the empty arena, then we set the learning rate to zero and the algorithm stops updating the background so it is ready to locate reliably the position of the animal with a frame rate of 20 Hz. LibSerial library is used to communicate with the AU during the task sending symbolic codes and changing the state of the AU in the OC chamber. We used Tkinter library to write the initial GUI to set the experimental parameters. The behavioral sequence is outlined in Figure 2B. Virtually the chamber can be divided into two sections: the anterior part that contains the interface between the mouse and the computer, and the posterior side that is designed as an active area to activate the trials. If the mouse remains in the active area for a given amount of time (1.5 s) the trial is triggered. At this stage a visual stimulus is shown on the display and the system waits for animal response. When the mouse touches one of the two buttons, an auditory feedback is produced, with a tone that varies depending on whether the answer is correct  Open Source Tools and Methods (3300 Hz) or wrong (2700 Hz). In case of correct answer, a drop (7 ml) of water with 1% condensed milk is released.
Implementation of an LCD screen As a proof of principle ofcustomizability, we added a version of the OC chamber that is able to show more complex visual stimuli. This version includes an edit of the frontal wall that can host a TFT monitor (Kookye 3.5" for RPI3) and a folder (LCD_oc_chamber) containing code that runs on Psychopy2 (Peirce, 2008), a Python package dedicated to behavioral experiments. This configuration allows to show RGB images as visual stimuli.

Code accessibility
The code described in the paper is freely available online at https://github.com/raffaelemazziotti/oc_chamber. The code is also available as Extended Data 1.

The OC protocol Familiarization
This phase is conducted by placing each animal in the OC box for three sessions of 10 min, spaced by at least 2 h between each other. During this phase, a liquid reward, coupled with the "correct" tone is provided manually whenever the mouse is in the active area, in this way the animal learns where to find the reward and associate it with the tone.

Shaping (3 d)
The visual stimulus is introduced (Fig. 2C). It consists of two bright (0.9 cd/m 2 ) blue (465-475 nm) dots (5 mm) that appear above the two buttons. The mouse needs to touch one of the two buttons to obtain the reward.

Operant task (OT, 5 d)
During this phase, only one dot appears, identifying the correct button. If the mouse touches the correct button, the correct tone is reproduced and the animal receives the reward. If the mouse touches the wrong button, the "wrong" tone is reproduced and no liquid reward will be delivered. This procedure is shown in Figure 2C. The sequence of stimuli is balanced so that the mouse sees each case the same number of times. In order to prevent perseveration with the same answer, during the first 2 d we adopted an assisted procedure (Fig. 2D): the first stimulus presented is random, if the mouse produces the correct answer the following stimulus is randomized, in case of wrong answer instead, the system repeats the same stimulus until the mouse gives the correct answer three times.

Follow-ups
In order to test the ability of the mouse to recall the task, we tested animals in different follow-ups, respectively, at 6 d, 27 d, three months, and four months approximately after the end of OT. For each recall sessions we tested mice once per day.

Data analysis and statistics
Data processing is performed using Python and the statistical analysis (Table 2) using GraphPad Prism 7. To analyze mouse tracking data the arena is virtually divided into 256 (16 Â 16) bins and raw exploration is z-scored to obtain relative exploration measures. To quantitatively test whether the mouse preferentially explores some of the bins, we constructed a resampled binned exploration matrix representing chance level for each bin, randomly permuting each animal exploration matrix for 100 times. The software Fritzing was used to draw the wiring diagram of the electrical components. We used Rhino 6 to draw the exploded version of the model.

Behavioral performance
To test our system ability to detect learning curves, we trained mice as depicted in the protocol in Figure 3B. Table 2 reports all statistical analysis. In shaping phase, the average number of trials (TR) progressively increases over time for all the subjects and specifically the third day, we detect a significant increase compared with the first day. Moreover, RTs and intertrial intervals (ITIs) showed a similar trend with a significant reduction of the RT starting from the second day (Fig. 3A). This indicates that already at day 2, the animals started to refine the sequence of actions necessary to trigger the stimulus and produce a response. Next, the results of the OT phase are shown in Figure 3C. The average TR continued to grow until day 3 of OT. After this day, the majority of the animals performed the maximum TR permitted in each session. Both RT and ITI showed a decrease with time. Indeed, RT and ITI dropped significantly during the first 2 d and reached a plateau by the third day. We observed a significant difference in the percentage of correct responses between the first and second day. In order to assess the retention of the test over time, we tested the same mice at different time points after the end of the OT. Accuracy remained stable during all time points tested; however, RT showed a more complex pattern: with an initial decrease, compared with the last day of the OT, followed by an increase at four months. Analyzing ITI, we detected an increase of the time between two trials at 27 d that remained higher at three and four months compared with the end of the OT (Fig. 3D). It is interesting to note that, since touch sensors are activated from all sides, some of the variability in timing performance could be explained by the development of different strategies to activate sensors.

Tracking analysis
In Figure 4A, tracking traces, of all the mice, are shown with corresponding heatmaps, averaged across animals (on the right) or days (bottom row), showing non-uniform exploration of the OC chamber during tasks. Pixels that were not significantly explored compared with randomly resampled uniform exploration values (p . 0.05) were set to 0. The reward area was the most visited place, as shown by both the animals and session average heatmaps. In the bottom half of the arena there are two significant exploration spots at the corners, that indicate a stereotyped strategy to activate the trial (Fig. 4B). Moreover, we analyzed the distance traveled by each animal inside the OC box during all the tasks. We found that, throughout the course of the shaping phase, there was a significant decrease in the total amount of distance traveled compared with the first day (Fig. 4). Conversely, during the OT phase, we detected no changes (p = 0.3672). Interestingly, we found a significant correlation between timing performances and the total distance moved during shaping (Fig. 4C), this suggests that ;25% (r 2 = 0.247) of the improvement in timing performance is explained by a reduction in the distance traveled and the response of the animal. In addition, no differences in the average speed were detectable during both shaping and OT. These results imply that the reduction of the RT is due to the optimization of the psychomotor sequence in realizing the task rather than to a general increase of velocity of the animal.

Discussion
Here, we described a fully 3D printable device that performs OC on freely moving mice while tracking the animal position in real time. We successfully trained 6 subjects, showing stereotyped learning curves that are highly reproducible across mice and reaching .70% of accuracy after 2 d of conditioning (Movie 1). This dataset demonstrate that this system is a valuable low-cost alternative to available state-of-the-art commercial devices, representing a good balance between performance, cost, and versatility. Performances detected by our system in three sessions per day (3.97 6 0.11 trial/min with an accuracy of 84.1 6 1.7%) are comparable with normative values detected in C57BL/6J and measured on an analogous two-alternative forced choice task performed once daily (Malkki et al., 2010). Although the LED display does not allow to design complex visual patterns required to perform image recognition and classification, visual stimulation is flexible enough to design simple tasks to test attention, learning, memory and other neuropsychological aspects of cognition (D'Ausilio, 2012;Escobar and Pérez-Herrera, 2015). The system is also easily customizable, as it is possible to add a LCD display (Movie 2) guided by extra Python libraries (e.g., Pygame or Psychopy). The overall cost of the chamber is around 160 e, but can be further substantially reduced using cheaper boards compared with AU and a Raspberry Pi. There are other low-cost alternatives for OC (Escobar and Pérez-Herrera, 2015;Francis and Kanold, 2017;O'Leary et al., 2018); however, the main strength of the present device is the high degree of reproducibility, since the model is standardized and can be downloaded to print the same structure with the same materials throughout different laboratories. Second, it can be customized in relation with specific experimental needs (e.g., very young animals). Lastly, different versions of the OC chamber can be tested and shared through on-line repositories, such as Thingiverse (https://www.thingiverse.com/) and NIH Print Exchange (https://3dprint.nih.gov/). Moreover, the OC chamber includes real time tracking of the mouse position, a feature that could be used as second phenotyping measure of anxiety or stereotyped behaviors. Additionally it allows to analyze other aspects of behavior, such as inhibitory control (Munakata et al., 2011). For example, by increasing the time required to trigger a trial, it is possible to measure impulsivity or reproduce neuropsychological tests used on humans like delayed gratification or stop signal tasks (Pinkston and Lamb, 2011;Furlong et al., 2016). It is also plausible to couple the procedure with physiological recordings in freely moving conditions such as imaging techniques (e.g., fiber photometry) and electrophysiology. Thanks to the general-purpose input/output ports (GPIO) of both AU and Raspberry Pi boards, high precision synchronization of physiological recordings with behavioral events is accurately integrated within experimental recording paradigms. The simplicity and modularity of the apparatus can