I want to bring data from controller through a Modbus link. The flow is 0-101000 SCFM and the register is scaled for 0-102.4%.

The option would report 100 percent as 0x0FA0. 0x0FFF would then correspond to 102.375 percent.

If I convert this 0X0FFF in numeric then it will be a 32716.

How can I scale this Modbus address in 30001 series for 0 - 100 % value with square root considered here?

 

y = mx + b

b = offset (zero in your case)
m = scale factor.
x = original value
y = scaled value

if 100% = 101000, then m = x/y, or 32716 / 101000, or m = 0.323920792. In other words, multiply your original value by 0.323920792, or divide it by 3.087174471. If you don't have floating point math on your device, then you will have to use an integer scale factor (e.g. 3) and 100% will be slightly lower (not 0x0FA0). If you don't have division but have shift right/left, then you can use a right shift of 2 places (which is the same as divide by 4) and accept a lower full scale value.

If this doesn't answer your question, then you need to be clearer as to what your question is.