2.1.

Simulated Photoacoustic Datasets

To evaluate the performance of the developed application, three numerical datasets such as point targets, triangular shape, and vessel shape were generated using the k-Wave toolbox.⁴¹ Although in practice UST with a large active area is used, an ideal point detector was used for simulation. A circular scanning geometry with a scanning radius of 41 mm was used. The computational grid was $820\times 820$ ($0.1\mathrm{mm}/\text{pixel}$) with a perfectly matched bounding layer [Fig. 1(b)]. An ideal detector (2.25 MHz central frequency and 70% nominal bandwidth) was utilized for acquiring the PA data at 800 uniformly distributed positions around the sample, with 1500 steps at each position and a sampling frequency of 25 MHz. The speed of sound considered was $1500\mathrm{m}/\mathrm{s}$ and the medium chosen was acoustically homogeneous. The first numerical phantom consisted of five-point targets, as shown in Fig. 1(c). The distance between two adjacent points was 5 mm while the intermediate point was located at the scanning center. Figures 1(d) and 1(e) present the numerical phantom of the isosceles triangle and brain vasculature that mimics the venous sinuses of rat brain, respectively.

2.2.

Experimental Phantom Imaging

For experimental phantom imaging, two types of phantoms were used for validating the performance of the developed application, namely a point source phantom that was made up of pencil leads and a triangular phantom made up of horsehair. As shown in Fig. 1(f), a $\sim 532$-nm Q-switched Nd:YAG laser delivering 10 pulses per second with a pulse width of 5 ns was used as illumination source.⁴² An unfocused UST (Olympus NDT, V306-SU) with a central frequency of 2.25 MHz, active area of 13 mm, and fractional bandwidth of 70%, was used to scan the sample with a scanning time of 480 s. 4800 A-lines were obtained during the 8-min scanning process, and for each A-line, 1024 samples were captured with a sampling frequency of 25 MHz. Averaging was performed for every two A-lines so that the number of A-lines became 2400. The collected PA signals were amplified via a low signal noise amplifier (Olympus-NDT, 5072PR) and stored in a computer (IntelXeon, 3.7 GHz 64-bit processor, 16 GB RAM) using a DAQ card (GaGe, compuscope 4227).

2.3.

In Vivo Photoacoustic Imaging

The schematic diagram of the pulsed laser diode (PLD)-based PA imaging system, PLD-PAT system⁴³ that was used for in vivo imaging is shown in Fig. 1(g). The PLD produced $\sim 816\mathrm{nm}$ light with a pulse repetition rate of 2000 Hz and a pulse width of $\sim 107\mathrm{nm}$. A laser driving unit (LDU) was used to control the PLD. The LDU comprises a water-cooling unit, a low voltage power supply of 12 V (Voltcraft, PPS-11810), a variable high voltage power supply (Elektro-Automatik, EAPS 8160-04T) that can control the laser power, and a function generator (RIGOL, DG1022) that can control the pulse repetition rate. The laser beam homogenized by an optical diffuser was expanded over an area of $\sim 20{\mathrm{cm}}^2$. The laser irradiated on the sample surface met the requirements of American National Standards Institute safety limit.⁴⁴ Single-element unfocused UST (Olympus-NDT, V309-SU) was employed to collect PA waves, with a central frequency of 5 MHz, active area of 13 mm, and fractional bandwidth of 70%. A 45 deg acoustic reflector (F102, Olympus NDT) was placed on the transducer body. 600 A-lines were obtained by continuously moving the UST in a circle around the sample for a scan time of 12 s, with each A-line containing 1024 samples. The UST was driven by the stepper motor (SM), which was controlled by the computer. The collected A-lines were amplified by a 48 decibels (dB) low signal noise amplifier (mini-circuits, ZFL-500LN-BNC), followed by being stored inside the computer (Intel Xeon, 3.7 GHz 64-bit processor, 16 GB RAM) via a DAQ card (Spectrum, M2i.4932-Exp). The DAQ was synchronized with the laser irradiation by the functional generator. Sprague Dawley rats (weighing $90\pm 5\text{grams}$) were obtained from InVivos Pte. Ltd., Singapore for experiments. In vivo imaging was performed with one rat at a time, and the pre-imaging operations were the same for all rats. Before the imaging, each rat was anesthetized by a combination of ketamine ($100\mathrm{mg}/\mathrm{mL}$) and xylazine ($20\mathrm{mg}/\mathrm{mL}$), followed by hair removal by employing depilatory cream and the utilization of ocular gel on both eyes. A layer of ultrasound gel was applied to the rat scalp for better coupling. During the imaging process, 0.75% isoflurane and $1.0\mathrm{L}/\min$ oxygen were continuously supplied to keep the rat anesthetized. After the experiments, the rat was euthanized by the intraperitoneal administration of Valabarb (sodium pentobarbitone $300\mathrm{mg}/\mathrm{mL}$). All the animal experiments were performed under the guidelines of the Institutional Animal Care and Use Committee, Nanyang Technological University, Singapore (Protocol No.: A0331).

2.4.

Development of a Mobile-Platform-Based Algorithm

Python 3.9.5 was used for developing an Android-based application due to its several merits. Compared with another program, Android Studio, which is also commonly used for creating Android-based applications, Python is an interpreted language and can be used cross-platform. Many open-source libraries are available in Python, allowing users to flexibly call commands to accomplish certain tasks. Besides, users can call the functions in other languages like C and C++ to run the command more effectively since Python is extensible. In addition, to further improve the efficiency of the mobile-platform algorithm, we can employ Fourier-based or deep-learning-based reconstruction algorithms in the future, which can be more easily achieved in Python. A cross-platform Python framework, Kivy, was employed here for the development of the application. The codebase developed in this article mainly targets the Android system, though it can extensively target Linux, iOS, and Windows.

The process of developing the application was illustrated in [Fig. 2(a)]. Storage permission was first asked for when opening the application, followed by uploading the raw data stored on phones. The upload screen and the popup for uploading files were shown in Figs. 2(b) and 2(c), respectively. Two parameters, including the radius and angle step, were provided by users in the upload screen. During the image-forming process, the loading screen was shown as the transition screen. After a while, the beamforming process finished and the image-display screen was exhibited, where the reconstructed image and output parameters were displayed [Fig. 2(d)]. The output parameters contained the running time of the image-forming process and the signal-to-noise ratio (SNR) value of the image, which were shown at the bottom of screen. The reconstruction algorithm used here was a simple DAS beamformer, which defined the reconstructed image size as $250\times 250\text{pixels}$ ($0.1\mathrm{mm}/\text{pixel}$). The pixel size was smaller than both the spatial resolution values of the PLD-PAT system, which utilized a 5 MHz UST (resolution is $\sim 180\mu \mathrm{m}$), and the Nd:YAG-PAT system, which employed a 2.25-MHz UST (resolution is $\sim 380\mu \mathrm{m}$). In addition, optimization of the application was implemented by using effective functions, such as Enumerate and improving loading animation, aiming at shortening the time taken for reconstruction.

Fig. 2

(a) How to develop Android app in Python via Kivy. (b) Upload screen. (c) Popup for uploading raw data to the application. (d) Image-display screen on the Android phone (Huawei P20), which showed reconstructed image, reconstruction time, and the SNR value. (e) How to transfer Python code to mobile platform.

Although Kivy is cross-platform, packing the Python code into an executable program for the mobile platform is necessary. To run Python code on an Android phone, the code should be packaged into apk installer. There are several ways to finish packaging, including packaging via Python-for-Android (p4a), Kivy launcher, or Buildozer. In this work, we utilized Buildozer to package the Python code file into apk file since it was easily realized due to its integrated framework and can solve the dependencies of deploying to different platforms very well. In addition, while Buildozer debugging builds, the prerequisites were downloaded automatically, like p4a, Android SDK, and NDK, which indicated that Buildozer can further encapsulate p4a. However, Buildozer only compiled with Linux and macOS, and Windows is not supported. Before downloading Buildozer, a virtual machine or dual system was required to be installed for using Ubuntu, which was a Linux distribution based on Debian. After Buildozer was downloaded, an Android build could be initiated, leading to an apk file being created. By downloading apk into an Android phone, we can run the program on the phone. The whole process of transferring Python code into an executable mobile program was illustrated in Fig. 2(e).

2.5.

Image Quality Assessment

Several image quality matrices were used to assess the reconstructed image quality. First, SNR was calculated after beamforming and normalization, using the following equations:

3.1.

Image Reconstruction on Mobile Platform

Figure 3 depicts the reconstructed images of both numerical and experimental datasets on Huawei P20 using the DAS algorithm. The SNR values of Figs. 3(a)–3(h) was displayed in Fig. 4(b). The reconstructed images of the numerical phantom are shown in Figs. 3(a)–3(c), in which the SNR values are 48, 43.3, and 40 dB, respectively. The structures of the targets have been properly reconstructed, which were well matched with the ideal condition as shown in Figs. 1(c)–1(e).

Fig. 3

Reconstructed PAT images displayed on the mobile platform: images formed from (a) numerical point source data, (b) numerical triangle data, (c) numerical brain vessel data, (d) experimental point phantom data, and (e) experimental triangular phantom data. (f) In vivo brain vessel data, which were captured with 600 steps. (g) In vivo brain vessel data, which were downsampled to 300 steps. (h) In vivo brain vessel data, which were downsampled to 200 steps. The background region and a part of photoacoustic region (marked by blue solid box and blue dashed box) are shown in the bottom row.

Fig. 4

(a) Reconstructed time comparison among different datasets. (b) SNR comparison among different reconstructed images formed from different datasets. (c) PSNR comparison between images from twofold downsampled and threefold downsampled datasets; the images from original datasets were taken as reference images. (d) SSIM comparison between images from twofold downsampled and threefold downsampled datasets; the images from original datasets were taken as reference images. (e) Comparison of beamforming time on the mobile platform among different image-forming process. N, numerical dataset; E, experimental dataset. (f) Comparison of DAS image formation time on different mobile platforms; the data used here was in vivo original and downsampled data.

Although more A-lines were captured for experimental phantom imaging, the PA images formed from the phantom experimental dataset showed poorer quality compared with the PA images formed from the numerical dataset. This could be due to the noise added in the numerical datasets being lower than the actual experimental noise. The SNR values of Figs. 3(d) and 3(e) were 39.5 and 37.5 dB, respectively, around 8 dB lower than that of Figs. 3(a) and 3(b). The point targets were not well reconstructed, especially for the points that were far from the scanning center. This is expected, as the spatially variant tangential resolution due to the large aperture size of the detector used for experimental dataset is well known.^29,42 As for Fig. 3(e), the structures of the triangle targets were reconstructed properly though the endpoints was blurred and the background noise was higher relative to Fig. 3(b). Using PLD-PAT system, rat brain was imaged. In vivo PA signals were captured at 600 uniformly distributed positions and 1024 samples were acquired in each position, resulting in a data size of $600\times 1024$. The reconstructed images of the brain vessels were displayed in Fig. 3(f), with SNR of 37 dB. Both the transverse sinus and sagittal sinus were clearly observed, though some noises existed in the background.

To quantitatively evaluate the influence of data size on reconstruction image quality and time, six datasets were downsampled two times and three times in the number of positions around the circle. The reconstruction time for the original datasets and downsampled datasets was tested by five runs. Figure 4(a) shows the mean reconstruction time and standard deviation for each dataset, whereas Fig. 4(b) shows the corresponding SNR values of the reconstructed images. It can be found from Fig. 4(a) that for datasets of the same size (number of A-lines $x$ samples in each A-line), irrespective of whether it is simulated or experimental data and what type of simulated data, the time for PAT image reconstruction was similar. Such relationship can be more easily seen in Table 1. Here, the reconstruction time was the average reconstruction time used for the same size of datasets. When the original dataset was downsampled twofold, the reconstruction time was reduced by approximately half, whereas the SNR value was only decreased by $<4\mathrm{dB}$. Considering the limitation of only taking SNR as image quality metric, PSNR and SSIM were also employed to assess the quality of images. As shown in Fig. 4(c), the PSNR values for all twofold downsampled datasets were higher than 40 dB, except for in vivo data, which was still $\sim 40\mathrm{dB}$ (39.7 dB). This reveals that the difference between the image generated from original datasets and twofold downsampled datasets was relatively low in terms of the maximum image pixel values and the noise levels. Also, the SSIM values of the reconstructed images from the downsampled datasets were all not $<0.95$. A high degree of similarity between the reconstructed images and reference images was thus demonstrated. Minimum distortions of the image reconstructed from twofold downsampled datasets can be further verified by the PA image reconstructed from two-fold downsampled in vivo data [Fig. 3(g)]. A part of background region and a part of photoacoustic target region were enlarged (marked by blue solid box and blue dashed box), suggesting that the structures of the photoacoustic target were still properly reconstructed, and the background was clean with not much noise perceptible. The quantitative and qualitative results demonstrate that the twofold downsampling process significantly reduced computation time without significantly degrading the image quality.

Table 1

Relationship between the size of dataset (no. of A-lines × samples in each A-line) and reconstruction time.

The reconstruction time could further be reduced by the three-fold downsampling operation, which led to about three times decrease in computation time. However, the images formed from threefold downsampled data show visible degradation (streak artifacts), as the SNR value of these images dropped compared to the images formed from original data. Streak artifacts are quite common when the number of A-lines used for reconstruction is low. The degradation can be observed from PSNR and SSIM values. The PSNR of the images formed from threefold downsampled datasets were around 35 dB, except for numerical triangle dataset (29.7 dB) and numerical brain vessel dataset (38.6 dB). The difference between the images from threefold downsampled datasets and original datasets became more pronounced, implying a loss of information during the reconstruction process (due to low number of A-lines). Similarly, the SSIM of images declined to around 0.96 for simulated datasets except for numerical triangle dataset, around 0.93 for experimental phantom dataset, and even lower, 0.88 for in vivo dataset, which implied a lower degree of similarity between the reconstructed image and the reference image. However, the SSIM values higher than 85% still demonstrate the proper reconstruction of target structure, though some degradation was present (more pronounced in the background areas). It is worth noting that the SSIM value for the image from numerical triangle dataset is highest among the images formed from threefold downsampled datasets, though the PSNR value of that is the lowest. Such divergence could be due to the fact that the image had a high SSIM with the reference image though it has undergone some form of compression that has reduced the pixel values, resulting in a lower dynamic range and a lower PSNR. But the image still retained the essential structural features of the reference image. Figure 3(h) showed the reconstructed image from three-fold downsampled in vivo data in which the background artifact (streak artifact) was notably higher than Fig. 3(f), leading to an SNR decrease of more than 5 dB. But three-fold downsampling process still led to the PA images with comparable target structure to that of the PA images obtained from the original datasets, which was consistent with the SSIM value of the image.

These findings suggested that there was a tradeoff between image quality and reconstruction time, as larger-sized datasets led to images with higher quality but longer reconstruction time, whereas smaller-sized datasets resulted in lower-quality reconstructed images but reduced reconstruction time. Limited by the size of the screen and screen resolution, smartphones cannot display high-resolution images as clearly as desktops/workstations/laptops, indicating that the phone application can focus more on the time consumed for the image-forming process instead of image quality. Considering the influence of downsampling on reconstruction time and image quality, two-fold downsampling led to much less time spent on beamforming but still maintained the image quality without much loss, offering a balance between speed and quality. An additional step was done to verify that transferring the Python file into apk file would not influence the reconstructed image quality. We employed the same DAS algorithm and the same raw dataset in laptop Python to obtain the reconstructed images, for which the SNR values were the same as that of images reconstructed on mobile platforms.

3.2.

Reconstruction Time Comparison Between Mobile Platforms and Laptop

Figure 4(c) demonstrates the reconstruction time difference between the laptop and mobile platforms (Huawei P20) among different samples. The laptop used here was Lenovo Thinkbook 15, possessing 24 GB DDR4 memory and a processor of 10’th Gen Intel^®Core™i7. To evaluate the stability of the application, five runs have been tested on six datasets on both laptop and Huawei P20. The mean time and standard deviation were calculated for the reconstruction time. The datasets used here were maintained in their original size without any downsampling performed. As for the same dataset, the time taken for reconstruction in Huawei P20 was significantly higher than that on the laptop. Around 7-s differences were observed between the image reconstruction time on laptop and mobile platforms for all numerical datasets (800 steps × 1500 samples). For larger-size datasets (2400 steps × 1024 samples, for experimental phantom datasets), the time spent on reconstruction showed a huge difference between laptop and mobile platforms. Around 50 s was required for the laptop to form the PAT images, and the mobile application took an additional 30 s to fulfill the same beamforming process. The significant difference in the reconstruction time in the two platforms was mostly owing to the relatively less advanced processor of Huawei P20, which was released four years ago. In vivo brain vessel dataset comprised fewer DAQ points (600 points), resulting in significantly less time for DAS. Without performing downsampling on the raw dataset, there was about a 5-s reconstruction time difference between the two platforms. These results indicated that the mobile application possesses no significant running time difference from the laptop when performing the DAS for the small-sized datasets, whereas a significant time difference would be there between the two platforms for the large-sized datasets. Performing downsampling to the dataset can accelerate the image reconstruction, and the reconstructed image showed not many differences from the image reconstructed from the original large dataset. In addition, the standard deviation of five runs on the mobile platforms was lower than 1 s, suggesting the stability of the application.

3.3.

Reconstruction Time Between Different Mobile Platforms

Mobile phone performance varies significantly with different specifications, including central processing unit (CPU), GPU, and internal memory size. To measure the extent to which different specifications affect reconstruction time and verify that the application can be universally applied on Android platforms, we compared the running time on six different Android phones, as shown in Fig. 4(d). Their specification is presented in Table 2, and Samsung Galaxy S21+ is the most recent Android phone that was tested. In vivo brain vessel dataset was used here, with both the original (600-step) and downsampled (200-step) datasets. The first developed version of the application required $127.7\pm 8.4$ seconds to accomplish the image-forming process for the 600-step in vivo brain dataset on a Samsung Galaxy S9 phone. Owing to performing optimization operation including importing only the required modules of a library, employing functions with short execution time, and using smoother loading animation, the reconstruction time was reduced to $10.7\pm 0.0\mathrm{s}$.

Table 2

Specification of seven Android mobile platforms.

With similar release year and configurations, Huawei P20 and Samsung Galaxy S8+ completed image forming process within similar time for both datasets. 6.9 s and 20.2 s were needed for Huawei P20 to perform image reconstruction in 200-step and 600-step brain vessel data, respectively. Similarly, 6.5 s and 18.9 s were required for Samsung Galaxy S8+ to finish the beamforming process in both reduced-sized and original datasets. For the Samsung Galaxy series of smartphones with improved specifications, the reconstruction required less and less running time. Compared to Huawei P20, the newly released Samsung Galaxy S21+ possessed an advanced processor, with $>60\%$ decrease in time spent on the image-forming process. 2.4 s and 6.9 s were required for the reconstruction of downsampled dataset and the original dataset, respectively. The less running time revealed the possibility that the advanced mobile system owned similar or even superior performance in the image-forming process compared to the laptop.

In this work, the reconstructed image size was fixed at $250\times 250\text{pixels}$ ($0.1\mathrm{mm}/\text{pixel}$), which means that only the photoacoustic targets within a region of $2.5\times 2.5\mathrm{cm}$ can be reconstructed. However, the image size can be adjusted arbitrarily by modifying the parameters that control its height and width in the Python scripts. As for the dataset size, there is no limit as long as it does not exceed the maximum memory capacity of the phone. The development of an Android-based application for photoacoustic image reconstruction is expected to contribute significantly to the advancement of portable phone-based photoacoustic imaging systems. These systems have the potential to improve medical diagnostics and make healthcare more accessible to underserved populations. However, building such systems has been a challenge due to the difficulty of integrating high energy light sources or the ultrasound detectors within the phone. Advancement of LED light sources could accelerate the development of mobile PA imaging system. Mobile based ultrasound imaging systems are already developed. Potentially one can modify such a system to get the raw channel ultrasound radio frequency (RF) data (not a trivial task though), as already beamformed RF data will not be useful for PA imaging. On the other hand, controlling the hardware of an external laser (or other potential light source) from a phone, DAQ, digitization and transfer into a mobile phone might be possible but has not been demonstrated yet. So, there are few different paths through which it might be possible in the future to develop the full PAT imaging system on a mobile. Although on the image acquisition side not much work has been done for a mobile-based PAT system, it might be feasible to do so soon. Our future work will focus on not only the mobile based reconstruction but also developing mobile based PAT imaging system.