DLC processor file#
The dlc processor file is a python script that is loaded into the dlclive-gui. This file first establishes a socket with the vr4mice software. Then every time the camera captures an image of the mouse in the arena, it processes the pose of the animal and computes the mean position of its head on each frame in x, y, alongside the head angle and heading direction of the body. In addition to this the dlc processor generates an oscillating signal that can be sent to the unity game and rendered as a black and white square in the top right hand corner of one of the computer monitors. This signal can be read using a photodiode that is attached to the screen via a teensy and is read on a separate thread within the dlc processor and time stamped (please note that this is a separate teensy to the one that is used to deliver water rewards). This can be used either for latency testing or for synchronization with a recording device. PLEASE USE THE dlc_inference_w_pd class.
class dlc_inference_w_pd(Processor):
def __init__(self, com = "COM3", baudrate=9600, signal_delay = 10, signal_type ="pulse", freq =5, use_teensy = 1):
com: Str, the com port for the teensy, default isCOM3baudrate: Int, the baudrate that the teensy is running at, default is9600signal_delay: Int, the delay between starting the game and the oscillating signal starting in seconds. This allows for the game to start before the signal starts being generated.signal_type: Str, options:pulse= periodic square wave pulsepulse_geo= square wave with geometric jitter (recommended for non-periodic TTL)sin= sine wave outputflip= toggle every frame (caution: can miss frames!!!) Default ispulse_geo.
freq: Int, the frequency of the square pulse or sine wave. default is5use_teensy: Bool: If the photodiode teensy is connected this should be set toTruefor recording the photodiode signal. If the photodiode teensy is not connected you can set this toFalse. If set toFalsethe dlc processor will still send a signal but this will not be recorded using the photodiode.
Photodiode teensy#
when the script is initialized and the use_teensy parameter is set to True, the script starts reading from the teensy on a separate thread:
if use_teensy == 1:
self.teensy = TeensyLatency(com, baudrate=baudrate)
print("using_teensy")
The process function#
The process function takes the dlc pose and computes the head x,y positions, head_angle and heading direction of the body and then send these via a socket to the vr4mice software.
def process(self, pose, **kwargs):
#print(pose.keys())
xy = pose[:, :2]
conf = pose[:, 2]
# compute head position using a weighted average of the head keypoints
head_xy = xy [[0, 1, 2, 3, 4, 5, 6, 26],:]
head_conf = conf [[0, 1, 2, 3, 4, 5, 6, 26]]
center = np.average(head_xy, axis=0, weights=head_conf)
# compute head angle and body heading direction
body_axis = xy[7] - xy[13] # tail_base -> neck
body_axis /= sqrt(np.sum(body_axis ** 2))
head_axis = xy[0] - xy[7] # neck -> nose
head_axis /= sqrt(np.sum(head_axis ** 2))
cross = body_axis[0] * head_axis[1] - head_axis[0] * body_axis[1]
sign = copysign(1, cross) # Positive when looking left
try:
head_angle = acos(body_axis @ head_axis) * sign
except ValueError:
head_angle = 0
#Generate synchronization signal
self.curr_time = time.time()
self.curr_signal = self.get_signal(curr_time = self.curr_time, st=self.start_time,
freq = self.signal_freq,
delay=self.signal_delay,
signal_type=self.signal_type)
self.curr_step + self.curr_step + 1
heading = atan2(body_axis[1], body_axis[0])
heading = degrees(heading)
vals = *center, heading % (360), head_angle, self.curr_signal
# add data for saving
self.center_x.append(vals [0])
self.center_y.append(vals [1])
#print("center_y: ", vals [1], ", center_x: ", vals [0])
self.heading_direction.append(vals [2])
self.head_angle.append(vals [3])
self.time_stamp.append(self.curr_time)
self.step.append(self.curr_step)
self.signal.append(self.curr_signal)
self.frame_time.append(kwargs ["frame_time"])
# send the data and time stamp via the socket
self.conn.send([time.time(), vals [0], vals [1], vals [2], vals [3], vals [4]])
return pose
Save function#
At the end of the experiment when the experimenter stops the recording and hits save video in the dlclivegui window the data is then saved using this function:
def save(self, file=None):
save_code = 0
if file:
print(file)
try:
save_dict = self.save_latency_data()
print(save_dict)
pickle.dump(
save_dict,
open(file, "wb"),
)
save_code = 1
except Exception:
save_code = -1
return save_code
def save_latency_data(self):
self.teensy.close_serial()
save_dict = dict()
save_dict ["start_time"] = np.array(self.start_time)
save_dict ["frame_time"] = np.array(self.frame_time)
save_dict ["time_stamp"] = np.array(self.time_stamp)
save_dict ["step"] = np.array(self.step)
save_dict ["signal"] = np.array(self.signal)
save_dict ["photodiode_read"] = np.array(self.teensy.input_data)
save_dict ["photodiode_time"] = np.array(self.teensy.input_data_time)
save_dict ["x_pos"] =np.array(self.center_x)
save_dict ["y_pos"] = np.array(self.center_y)
save_dict ["heading_direction"] = np.array(self.heading_direction)
save_dict ["head_angle"] = np.array(self.head_angle)
return(save_dict)
This data gets saved into *_PROC file which can be loaded as a pickle.
parameters:
start_time: Float, the time that the script was startedframe_time: Vector, the time stamp that for when the cameratime_stamp: Vector, time stamp for when data was sent through the socketstep: Vector, the step number containing how many times the process function was calledsignal: Vector, the generated oscillating signalphotodiode_read: Vector, the recorded signal from the photodiodephotodiode_time: Vector, the time stamps for each of the reads from the photodiodex_pos: Vector, the x position of the head (pixels)y_pos: Vector, the y position of the head (pixels)heading_direction: Vector, the heading direction of the body of the mouse (degrees)head_angle: Vector, the angle of the head relative to the body of the mouse (degrees)