Uncalibrated photometric stereo is a formidable challenge task in the field of 3D vision, aiming to reconstruct the surface normal of an object when its shape, material reflectance, and light conditions are all unknown. At present, it remains difficult to address when dealing with more general materials (e.g., anisotropy) with complex reflectance and objects with significant shadows. In addition, the presence of generalized bas-relief ambiguity, which refers to the inherent ambiguity between shape and light, further compounds the challenges of uncalibrated photometric stereo. To overcome these limitations, we propose a lightweight unsupervised neural inverse rendering architecture, called General Material Shadow-Neural Inverse Rendering (GMS-NIR), which can effectively solve the uncalibrated photometric stereo problem by combining a learnable general material reflectance model and a shadow rendering model. We design an optimization strategy that allows GMS-NIR to jointly optimize the surface normal, light direction, and light intensity in an unsupervised manner by utilizing backpropagation to minimize rendering errors. Thorough experiments with diverse real-world datasets affirm the superior performance of our approach compared with alternative uncalibrated photometric stereo methods. GMS-NIR’s ability to handle a variety of materials and complex object shapes while accurately reconstructing surface normal makes it a promising advancement in the fields of computer vision and 3D surface reconstruction.