MV3d
pip3 install PyQt5 sudo apt-get install vtk6 tcl-vtk python-vtk python3-tk ffmpeg
pip3 install mayavi pip3 install matplotlib pandas opencv-python pyyaml
boston didi team
- bostondiditeam
- nepal
- zxf8665905 : 추천
- lihua213#1
- lihua213#2: Old version
leeyevi
hengck23
https://github.com/hengck23/didi-udacity-2017
mv3d_ros_interface
- jinbeibei(??) :cpp
0. 환경 준비
- install tensorflow-gpu and CUDA.
- A Nvidia GPU card with computation capability
- ubuntu (* Cuda7.5에 맞는 버젼은 14.04임)
- CUDA (*environment_gpu.yml상 버젼 =7.5)
- cuDNN
- Download cuDNN v5.1 for CUDA 7.5 : Runtime lib., 소스설치방법
- Python3.5 for MV3D related code
- Tensorflow-GPU(version>1.0)
- Python2.7 for ROS related script
# https://github.com/adioshun/gitBook_DeepDrive/blob/master/papermultiview-3d-cnn/environment_gpu.yml
# conda env create -f environment_gpu.yml --name mv3d_p3_gpu
conda create -n python35 python=3.5
conda install tensorflow-gpu opencv3 shapely scikit-learn keras Cython matplotlib simplejson numba
pip install easydict
0.1 GPU용으로 설정 변경
src/net/lib/setup.py and src/lib/make.sh : "arch=sm_37" #Google Cloud GPU Tesla K80
# Which CUDA capabilities do we want to pre-build for?
# https://developer.nvidia.com/cuda-gpus
# Compute/shader model Cards
# 6.1 P4, P40, Titan X so CUDA_MODEL = 61
# 6.0 P100 so CUDA_MODEL = 60
# 5.2 M40
# 3.7 K80
# 3.5 K40, K20
# 3.0 K10, Grid K520 (AWS G2)
# Other Nvidia shader models should work, but they will require extra startup
# time as the code is pre-optimized for them.
CUDA_MODELS=30 35 37 52 60 61
test
import tensorflow as tf
sess = tf.Session()
print(tf.__version__) # version more than v1.
1. 데이터 다운로드
The KITTI Vision Benchmark Suite Raw Data
1.9 수정 필요
A.
data/raw/kitti/경로 밑에 데이터 위치tracklet_labels.xml파일은2011_09_26_drive_0001_sync하위 폴더에 위치
B.
src/kitti_data/pykitti/tracklet.py L289에서 다운 받은 파일명으로 변경
DEFAULT_DRIVE = '2011_09_26_drive_0001'
2. ./src/make.sh
2.1 실행 방법
cd src
source activate didi
sudo chmod 755 ./make.sh
./make.sh
conda create -n python27 python=2.7
아래 [2.2]를 직접 실행 하는것 추천
2.2 실행시 진행 내용
#- `python ./net/lib/setup.py build_ext --inplace` : Fast R-CNN (MS)
#- 'bash ./net/lib/make.sh` : building psroi_pooling layer
#- build required .so files
ln -s ./net/lib/roi_pooling_layer/roi_pooling.so ./net/roipooling_op/roi_pooling.so
ln -s ./net/lib/nms/gpu_nms.cpython-35m-x86_64-linux-gnu.so ./net/processing/gpu_nms.cpython-35m-x86_64-linux-gnu.so
ln -s ./net/lib/nms/cpu_nms.cpython-35m-x86_64-linux-gnu.so ./net/processing/cpu_nms.cpython-35m-x86_64-linux-gnu.so
ln -s ./net/lib/utils/cython_bbox.cpython-35m-x86_64-linux-gnu.so ./net/processing/cython_bbox.cpython-35m-x86_64-linux-gnu.so
[에러] nvcc 못 찾을경우
- 절대 경로로 수정 후 실행
[에러] arning: calling a constexpr __host__ function from a __host__ __device__ function is not allowed.
make.sh파일에 아래 flag--expt-relaxed-constexpr추가
if [ -d "$CUDA_PATH" ]; then
nvcc -std=c++11 -c -o roi_pooling_op.cu.o roi_pooling_op_gpu.cu.cc \
-I $TF_INC -D GOOGLE_CUDA=1 -x cu -Xcompiler -fPIC $CXXFLAGS \
-arch=sm_37 --expt-relaxed-constexpr
3. Preprocess data (./src/data.py)
kitti기준, didi data 이용시
utils/bag_to_kitti실행 필요
- MV3D net 학습시 필요한 입력 데이터 생성
- Lidar bird eye view features
- Lidar front view features
- RGB image
- Ground Truth label
- Ground bounding box coordinate
- time stamp
./data/preprocessing/kitti/
- gt_boxes3d :npy
- gt_box_plot : png
- gt_labels : npy
- rgb : png
- top : .npy.npz
- top_image : png
 |
 |
|---|---|
3.9 수정 필요
A. data.py 수정
작업 환경
#data.py
if config.cfg.USE_CLIDAR_TO_TOP:
SharedLib = ctypes.cdll.LoadLibrary('/workspace/mv3d/src/lidar_data_preprocess/'
'Python_to_C_Interface/ver3/LidarTopPreprocess.so')
#if config.cfg.USE_CLIDAR_TO_TOP:
# SharedLib = ctypes.cdll.LoadLibrary('/home/stu/MV3D/src/lidar_data_preprocess/'
# 'Python_to_C_Interface/ver3/LidarTopPreprocess.so')
B. NameError: name 'MATRIX_Mt' is not defined
# /MV3D/src/net/processing/boxes3d.py 상단에 추가
# ./src/config.py L126 참고
#rgb camera
MATRIX_Mt = ([[ 2.34773698e-04, 1.04494074e-02, 9.99945389e-01, 0.00000000e+00],
[ -9.99944155e-01, 1.05653536e-02, 1.24365378e-04, 0.00000000e+00],
[ -1.05634778e-02, -9.99889574e-01, 1.04513030e-02, 0.00000000e+00],
[ 5.93721868e-02, -7.51087914e-02, -2.72132796e-01, 1.00000000e+00]])
MATRIX_Kt = ([[ 721.5377, 0. , 0. ],
[ 0. , 721.5377, 0. ],
[ 609.5593, 172.854 , 1. ]])
A. ./src/config.py
#if __C.DATA_SETS_TYPE=='test':
# __C.DATA_SETS_DIR = osp.abspath('/home/stu/round12_data_test')
if __C.DATA_SETS_TYPE=='test':
__C.DATA_SETS_DIR = osp.abspath('/workspace/mv3d')
B. roi_pooling.so을 심볼릭이 아닌 파일로 대체
이후에도 같은 문제가 발생 하므로 [C]방법 추천
cd ./src/net/roipooling_op
mv roi_pooling.so roi_pooling.so~
cp ../../net/lib/roi_pooling_layer/roi_pooling.so ./
C. roi_pooling.so 수정 버젼 다운 로드
- 다운로드 roi_pooling.so
- chmod +x roi_pooling.so
4. train.py
[에러] "tensorflow.python.framework.errors_impl.NotFoundError: YOUR_FOLDER/roi_pooling.so: undefined symbol: ZN10tensorflow7strings6StrCatB5cxx11ERKNS0_8AlphaNumES3"
- it is related to compilation of roi_pooling layer.
- A simple fix will be changing "GLIBCXX_USE_CXX11_ABI=1" to "GLIBCXX_USE_CXX11_ABI=0" in "src/net/lib/make.sh" (line 17)
OR Download and replace the .so with following file :[Download], CUDA 8.0, Python 3.5
[에러] NameError: name 'data_splitter' is not defined
user this version of train.py: for python2
[에러] "module 'tensorflow.python.ops.nn' has no attribute 'convolution'"
conda list | grep tensorflow후 tensorflow(cpu & gpu) 버젼을 1.0 이상으로 변경
File Structure
├── data <-- all data is stored here. (Introduced in detail below)
│ ├── predicted <-- after prediction, results will be saved here.
│ ├── preprocessed <-- MV3D net will take inputs from here(after data.py)
│ └── raw <-- raw data
├── environment_cpu.yml <-- install cpu version.
├── README.md
├── saved_model <--- model and weights saved here.
├── src <-- MV3D net related source code
│ ├── config.py
│ ├── data.py
│ ├── didi_data
│ ├── kitti_data
│ ├── lidar_data_preprocess
│ ├── make.sh
│ ├── model.py
│ ├── mv3d_net.py
│ ├── net
│ ├── play_demo.ipynb
│ ├── __pycache__
│ ├── tracking.py <--- prediction after training.
│ ├── tracklets
│ └── train.py <--- training the whole network.
│── utils <-- all related tools put here, like ros bag data into kitti format
│ └── bag_to_kitti <--- Take lidar value from ROS bag and save it as bin files.
└── external_models <-- use as a submodule, basically code from other repos.
└── didi-competition <--- Code from Udacity's challenge repo with slightly modification, sync with Udacity's new
updates regularly.
Related data are organized in this way. (Under /data directory)
├── predicted <-- after prediction, results will be saved here.
│ ├── didi <-- when didi dataset is used, the results will be put here
│ └── kitti <-- When kitti dataset used for prediction, put the results here
│ ├── iou_per_obj.csv <-- What will be evaluated for this competition, IoU score
│ ├── pr_per_iou.csv <--precision and recall rate per iou, currently not be evaluated by didi's rule
│ └── tracklet_labels_pred.xml <-- Tracklet generated from prediction pipeline.
├── preprocessed <-- Data will be fed into MV3D net (After processed by data.py)
│ ├── didi <-- When didi dataset is processed, save it here
│ └── kitti <-- When Kitti dataset is processed, save it here
│ ├── gt_boxes3d
│ └── 2011_09_26
│ └── 0005
| |___ 00000.npy
├ |── gt_labels
│ └── 2011_09_26
│ └── 0005
| |___ 00000.npy
| ├── rgb
│ └── 2011_09_26
│ └── 0005
| |___ 00000.png
| ├── top
│ └── 2011_09_26
│ └── 0005
| |___ 00000.npy
| └── top_image
| └── 2011_09_26
| └── 0005
| |___ 00000.png
└── raw <-- this strictly follow KITTI raw data file format, while seperated into didi and kitti dataset.
├── didi <-- will be something similar to kitti raw data format below.
└── kitti
└── 2011_09_26
├── 2011_09_26_drive_0005_sync
│ ├── image_02
│ │ ├── data
│ │ │ └── 0000000000.png
│ │ └── timestamps.txt
│ ├── tracklet_labels.xml
│ └── velodyne_points
│ ├── data
│ │ └── 0000000000.bin
│ ├── timestamps_end.txt
│ ├── timestamps_start.txt
│ └── timestamps.txt
├── calib_cam_to_cam.txt
├── calib_imu_to_velo.txt
└── calib_velo_to_cam.txt
0. setup the python environment.
#Google cloud GPU Tesla K80
#Ubuntu 16.4
conda create -n python2_gpu python=2.7
conda install -y numpy Cython tensorflow-gpu matplotlib scikit-learn PIL
pip install easydict opencv_python pyyaml mayavi
conda install -c anaconda cudatoolkit=7.5
OR
1. git clone
git clone --recursive https://github.com/leeyevi/MV3D_TF.git
2. Downloads KITTI object datasets.
# cd /workspace/MV3D/data/KITTI/object
wget http://kitti.is.tue.mpg.de/kitti/data_object_image_2.zip
wget http://kitti.is.tue.mpg.de/kitti/data_object_image_3.zip
wget http://kitti.is.tue.mpg.de/kitti/data_object_velodyne.zip
wget http://kitti.is.tue.mpg.de/kitti/data_object_calib.zip
wget http://kitti.is.tue.mpg.de/kitti/data_object_label_2.zip
/workspace/MV3D/data/KITTI/object/{testing/training}/lidar_bv 폴더 생성
3. Make Lidar Bird View data
- change the root_dir in
read_lidar.pyfile
Build the Cython modules
cd $MV3D/lib make
# edit the kitti_path in tools/read_lidar.py
# then start make data
python tools/read_lidar.py
./train_net.py --device gpu --device_id 0 --weights /workspace/MV3D_TF/data/pretrain_model/VGG_imagenet.npy --imdb kitti_train --iters 50001 --cfg /workspace/MV3D_TF/experiments/cfgs/faster_rcnn_end2end.yml --network MV3D_train
[에러] undefined symbol: ZN10tensorflow7strings6StrCatB5cxx11ERKNS0_8AlphaNumES3"
#/workspace/MV3D_TF/lib/make.sh
#g++ -std=c++11 -shared -o roi_pooling.so roi_pooling_op.cc \
g++ -std=c++11 -shared -D_GLIBCXX_USE_CXX11_ABI=0 -o roi_pooling.so roi_pooling_op.cc \
roi_pooling_op.cu.o -I $TF_INC -D GOOGLE_CUDA=1 -fPIC $CXXFLAGS \
-lcudart -L $CUDA_PATH/lib64