Glusterfs fuse client consuming high memory - memory leak

Question

I have set up my glusterfs cluster as Striped-Replicated on GCP Servers, but I am facing memory leak issue with Glusterfs fuse client. Its memory consumption is increasing every day.

Both glusterfs server and glusterfs fuse client is using the latest version(Client-4.1.5, Server- 4.1), but the below process is consuming high memory on the client servers.

glusterfs --fopen-keep-cache=off --volfile-server=gluster1 --volfile-id=/+ 

Every day I can see that memory consumption of the above process is increasing, a temporary fix is to unmount and kill the process.

In my glusterfs client server, I am getting 'Stale File Handle error'. Not sure this error caused the memory leak.

[2018-10-21 04:08:46.921985] W [fuse-bridge.c:1201:fuse_setattr_cbk] 0-glusterfs-fuse: 3705309: SETATTR() /Production/example.com/wp-content/cache/wpsol-cache/4bd4f0bf132901ecb17261f388864fd3 => -1 (Stale file handle)

Also, my glusterfs server is using default settings. Please help if there is any patches or fix.

Comments

what is the gluster version you are using?

---

For server 4.1.4, client(Fuse) is using 4.1.5. Both are having latest version.

---

try downgrading it to 

3.12.6

---

@priyaj, Why? is there any dependency problem with the latest versions?

---

Did you get this error before or after unmounting and killing the process?

---

This problem was initially found in 3.12 version itself but this was fixed. You can try using the 3.12 version.

---

try putting glusterfs inside cgroup with memory limit according to your resource availability using systemd and let me know if it works..

---

@priyaj,   I think the problem already exists with the old version as well. Is there any patches available ?

---

Are you sure that downgrading, also are you referring to glusterfs fuse version or server version?

---

@Jithin, yess try downgrading to either 3.12.8 or 3.12.6 as the issue was sorted in these versions, seems more promising.

---

Are u referring to server version or client version ?

---

@vardhan yes there are some unresolved bugs with the version 3.12.x and above

---

I'm referring the client version here @Jithin.

---

actually the problem was for high consumption of RAM but it was fixed in version 3.12.2 but again was taken care in 3.12.6

---

Thank you, let me downgrade it.

---

I think 3.12.6. is not available to download, can I use 3.12.15 ? I am using ubuntu client machines.

---

Try 3.12.8 if .6 is not possible.

---

@kalgi, it is also not available

---

@kalgi, It is also not available
Available versions are .15, .12, .5

Answer 1

'Stale File Handle error' occurs when the directory is deleted. So basically the pointer your shell has to that particular inode does not exist. It's just a way system treats the invalid inode pointers. In your case, I think this error occurs because you're unmounting and killing the process.

Comments

Okay, that is inode issue. We used unmounting to temporarily fix the load issue

Answer 2

@Kalgi, you mean the backend error, yes am continuously getting this error in backend

Answer 3

Heyy @Jithin,

Try taking statedumbs from different intervals and post your readings.