Transforming Data¶
One useful element of AVstack is the marriage between data and calibration. You can only imagine the frustration-savings! Let's walk through a few examples where this is useful.
import os
import avstack
import avapi
import numpy as np
%load_ext autoreload
%autoreload 2
data_base = '../../data'
obj_data_dir_k = os.path.join(data_base, 'KITTI/object')
raw_data_dir_k = os.path.join(data_base, 'KITTI/raw')
obj_data_dir_c = os.path.join(data_base, 'CARLA/ego-lidar')
KSM = avapi.kitti.KittiScenesManager(obj_data_dir_k, raw_data_dir_k, convert_raw=False)
KDM = KSM.get_scene_dataset_by_index(scene_idx=0)
CSM = avapi.carla.CarlaScenesManager(obj_data_dir_c)
CDM = CSM.get_scene_dataset_by_index(0)
DM = KDM # let's use KITTI for this one
/home/spencer/.cache/pypoetry/virtualenvs/avstack-docs-l0eE3ZqO-py3.10/lib/python3.10/site-packages/nuimages/nuimages.py:668: SyntaxWarning: "is not" with a literal. Did you mean "!="? if annotation_type is not 'none':
Jupyter environment detected. Enabling Open3D WebVisualizer. [Open3D INFO] WebRTC GUI backend enabled. [Open3D INFO] WebRTCWindowSystem: HTTP handshake server disabled. The autoreload extension is already loaded. To reload it, use: %reload_ext autoreload
Representing Objects in Different Sensor Frames¶
All objects in AVstack are specified relative to a reference coordinate frame. That means we can easily acquire object representations in different coordinate frames - the work is performed under-the-hood. The following show the same ground truth object in different coordinate reference frames.
frame = 0
objects_cam = DM.get_objects(frame=frame, sensor='main_camera')
objects_lid = DM.get_objects(frame=frame, sensor='main_lidar')
print(objects_cam[0].box3d, '\n')
print(objects_lid[0].box3d)
Box3D=[h: 2.17, w: 1.71, l: 4.33] x (x: -8.35 y: 2.09, z: 24.98) q: <class 'avstack.geometry.datastructs.Attitude'> - quaternion(0.49074671, -0.48384659, -0.50841785, 0.51630556), ReferenceFrame level 2, x: [0. 0.06 0.905], q: quaternion(0.49951805, 0.49674862, -0.50538759, 0.49830303), v: [0. 0. 0.], acc: [0. 0. 0.], ang: quaternion(1, 0, 0, 0) with reference: ReferenceFrame level 2, x: [0. 0.06 0.905], q: quaternion(0.49951805, 0.49674862, -0.50538759, 0.49830303), v: [0. 0. 0.], acc: [0. 0. 0.], ang: quaternion(1, 0, 0, 0) Box3D=[h: 2.17, w: 1.71, l: 4.33] x (x: 25.21 y: 8.60, z: -1.79) q: <class 'avstack.geometry.datastructs.Attitude'> - quaternion(0.02130213, 0, 0, -0.99977308), ReferenceFrame level 2, x: [-0.27 0. 0.985], q: quaternion(-0.99991581, 0.00461001, 0.00957989, -0.00743908), v: [0. 0. 0.], acc: [0. 0. 0.], ang: quaternion(1, 0, 0, 0) with reference: ReferenceFrame level 2, x: [-0.27 0. 0.985], q: quaternion(-0.99991581, 0.00461001, 0.00957989, -0.00743908), v: [0. 0. 0.], acc: [0. 0. 0.], ang: quaternion(1, 0, 0, 0)
We can verify these are nearly equivalent with:
obj_L1_cam = objects_lid[0].deepcopy()
obj_L1_cam.change_reference(objects_cam[0].reference, inplace=True)
# check exact equality
if obj_L1_cam.box3d != objects_cam[0].box3d:
print('Not exactly the same due to rounding errors!')
# check approximate equality
if obj_L1_cam.box3d.allclose(objects_cam[0].box3d):
print('But really close to the same!')
Not exactly the same due to rounding errors! But really close to the same!
Implicit Conversions¶
Some functions in AVstack are smart enough to know that they should convert under the hood when performing certain operations. One example is checking for bounding box overlap - overlap should be independent of the coordinate frame. A similar example is visualization.
# did we need to do conversion at all??
if objects_lid[0].box3d.allclose(objects_cam[0].box3d):
print('Conversion handled under the hood!')
# but again, appoximate
if objects_lid[0].box3d != objects_cam[0].box3d:
print('But again, not exact due to rounding errors')
Conversion handled under the hood! But again, not exact due to rounding errors
Let's check the bounding box overlap with the IoU (intersection-over-union) metric.
# checking bounding box overlap -- also implicitly converted!
objects_lid[0].box3d.IoU(objects_cam[0].box3d)
/home/spencer/.cache/pypoetry/virtualenvs/avstack-docs-l0eE3ZqO-py3.10/lib/python3.10/site-packages/numba/np/ufunc/parallel.py:371: NumbaWarning: The TBB threading layer requires TBB version 2021 update 6 or later i.e., TBB_INTERFACE_VERSION >= 12060. Found TBB_INTERFACE_VERSION = 12050. The TBB threading layer is disabled.
warnings.warn(problem)
1.0
Applying External Transformations¶
We can apply external transformations (rotations and translations) on any object. Rotations are quaternions and translations are 3-vectors. When applied to objects, in both cases, a new object is returned.
# -- rotation with a free-standing quaternion
q1 = avstack.geometry.transformations.transform_orientation([0, 20*np.pi/180, 0], 'euler', 'quat')
rot1 = avstack.geometry.Attitude(q=q1, reference=objects_lid[0].box3d.reference)
new_box1 = objects_lid[0].box3d.rotate(rot1, inplace=False)
if not new_box1.allclose(objects_lid[0].box3d):
print('We rotated! Original object unchanged.')
# -- can be inverted
new_box3 = new_box1.rotate(rot1.q.conjugate(), inplace=False) # inverted with transpose
if new_box3.allclose(objects_lid[0].box3d):
print('Back to the original with inversion')
We rotated! Original object unchanged. Back to the original with inversion
# -- translation with a vector
t1 = np.array([1, 2, -3])
new_box1 = objects_lid[0].box3d.translate(t1, inplace=False)
if not new_box1.allclose(objects_lid[0].box3d):
print('We translated! Original object unchanged.')
# -- translation with a Translation class
t2 = avstack.geometry.Vector(t1, reference=objects_lid[0].box3d.reference)
new_box2 = objects_lid[0].box3d.translate(t2, inplace=False)
if new_box1.allclose(new_box2):
print('Translation class and vector are the same!')
# -- can be inverted
new_box3 = new_box2.translate(-t2, inplace=False) # inverted with negative
if new_box3.allclose(objects_lid[0].box3d):
print('Back to the original with inversion')
We translated! Original object unchanged. Translation class and vector are the same! Back to the original with inversion
Ego-Relative Coordinates¶
Some datasets may have coordinates of the ego vehicle. This can be useful for relating the positioning of the ego to other objects in the scene, e.g., for vehicle-to-vehicle or vehicle-to-infrastructure communications. If we have ego coordinates, we can transform objects from global frame to local frame and vice-versa.
# # NOTE: in carla, sometimes the first few frames
# # don't have object (NPC) data while system is initialized
# frame = CDM.frames[10]
# objs_local = CDM.get_objects(frame, sensor='main_camera') # ego-relative, camera frame
# ego_pos = CDM.get_ego(frame)
# objs_global = [obj.local_to_global(ego_pos) for obj in objs_local] # global-relative, camera frame
# print(ego_pos)
# print(objs_local[0].position)
# print(objs_global[0].position)