Home » Other » Community Hangout » DBA's nightmares (Would it matter? LOL!?!)
DBA's nightmares [message #627693] Fri, 14 November 2014 10:52 Go to next message
Lalit Kumar B
Messages: 3174
Registered: May 2013
Location: World Wide on the Web
Senior Member
Okay, it's just another friday. But, not everyday it happens that a severity/priority 1 issue is raised(1 being highest).

A customer raises a serious concern about data being not populated after numerous attempts of executing an INSERT statement via SQL*Plus. Customer's DBA was involved before reaching us, and they had no luck. A call was setup immediately, and the discussion starts. Customer's DBA states that he checked every possible thing and that Oracle 12c has a bug, when data is inserted via SQL*Plus, it never shows up in the application. But, when it's done through the application using SAVE button it is fine. And then discussion goes into fine grained auditing etc. Everybody on the call shows their concern. My manager raises his eyebrow, staring at me as if he is concerned too(all fake, he doesn't even know what's going on). Customer claims it to be an issue with RAC, as they never encountered such issue before migrating to RAC.

I was forced to look into what went wrong with RAC DB settings. The SLA was already compromised because of the severity+priority of the issue(SLA for such cases is hardly few minutes). So, my manager goes wild, and for my sake, he said something which caught my ears. He said "Why not just force Oracle to commit to insert the data into database and blah blah blah....". It just clicked me about the session commit, I asked the customer whether he did a commit after the insert statement was executed. "What do you mean?" was his reply. I repeated myself, whether you did a commit or not? Asked the same question to their DBA, and then, there was a complete silence for almost a minute. I asked again on call, and then an unknown person speaks out. This unknown person, the actual developer(later we came to know he is an intern at customer's side) who raised the issue, speaks up. He said "I just did what Bob(his manager) said, i.e. to insert a new record and see if the reports are still screwed(for whatever reason). I lost my patience and pitched in the call, asked this STUPID developer if he issued COMMIT or not. Once again, after a silence of few seconds, there comes an answer, "NO".

And next I heard my manager shouting NO NO NO...., WTF... Next 2 minutes over the call was between my manager and the customer, and then call ended.

Okay, I think it's a long story, but WTF. I am still wondering who is going to lose their job, the developer or his DBA. I am still thinking if I did a great job asking the silly question about COMMIT. But, it happened so. I am back home now. Can't believe what happened today. Sigh!

Has anyone of you ever had such pathetic experience?

P.S. Today(November, 14th) is children's day celebrated in India. Folks on call who have kids, wasted their time on this useless issue. God bless them!

[Updated on: Fri, 14 November 2014 11:33]

Report message to a moderator

Re: DBA's nightmares [message #627708 is a reply to message #627693] Fri, 14 November 2014 20:07 Go to previous messageGo to next message
Barbara Boehmer
Messages: 9077
Registered: November 2002
Location: California, USA
Senior Member
I have heard worse. We generally started from the most basic: Is the computer plugged in? One person had difficulty determining whether it was plugged in or not. When asked why, it was explained that it was hard to see because the lights were out, due to the power outage. It is best just to remain calm and have patience. Nobody should get fired. Live and learn. Yelling and cussing only makes it worse. If you just explain things politely, you should be appreciated. We all do dumb things. Everybody has their own areas of expertise and should not be afraid to ask about things they don't know about. Imagine if he had not called and wasted a lot of time without figuring out the problem.


Re: DBA's nightmares [message #627715 is a reply to message #627708] Fri, 14 November 2014 23:55 Go to previous messageGo to next message
rleishman
Messages: 3728
Registered: October 2005
Location: Melbourne, Australia
Senior Member
I love that story; I've heard Barbara tell it before.

I don't have any good ones; just things like users entering data into Production and running reports in Test that don't contain their new data. Standard stuff.

When I saw the title, I was thinking of *real* DBA nightmares. Like having to use your backup and discovering it is corrupt. I have had two situations like this.

One was about 12 years ago. I was a Dev, not the DBA. I had been on-site about a week and we had a media failure. For reasons I won't go into (because I only heard them second hand, and they reflect poorly on one or more DBAs) the automated backup had been failing but not alerting (and nobody was checking). We had to put the database back together like a jigsaw puzzle, retaining the good tablespaces, restoring some tables from a month-old "good" backup, rebuilding indexes stored on the failed disk, and reloading data feeds (it was a warehouse) for all others. From diagnosis to recovery, it took two weeks of 12-hour shifts with teams rolling on and off to get sleep. Not a pleasant welcome to the site.

The other was much earlier in my career. I had about 12 months experience and was supporting an Oracle 5.1B application on IBM 286 DOS PC's by 2400 baud modem on 11 distributed sites around Australia. Being a single-session operating system, these were production systems sitting on an operator's desktop. EVERY single thing that happened on the system - every row entered, every batch job run, every report printed - was performed by the operator. This included backups. The PCs only had 60MB HDDs and there was no room on the disk for backups. I don't know whether Oracle had redo logs and point-in-time recovery back then, but we did backups via Export to 1.44MB floppy disks.
Anyway, somehow we lost some data; I don't remember how. It was an unimportant static table and I could get the rows from one of the other sites if I needed to, but thought it would be easier to use the backup. The Monday-Thursday backups only contained volatile tables, so I had to get the operator to get the most recent Friday backup. They did, and I started the DOS restore command to get it off the floppy disk. When it was done, it said "Insert Diskette #2 ... Hit any key to continue." I asked the operator over the phone to do just that. She didn't have diskette #2. About six months ago, the Friday backup - which used to fit on a single diskette - had started showing that message when it finished. She had never seen it before, assumed it was done, filed the disk in the safe and went home for the weekend. Oops. "Forget that backup", I told her. "Please get out TWO fresh diskettes; we're running a Friday backup NOW"

BTW - congrats on the (pending) milestone
/forum/fa/12289/0/

Ross Leishman
  • Attachment: barb8000.GIF
    (Size: 21.12KB, Downloaded 3485 times)
Re: DBA's nightmares [message #627720 is a reply to message #627715] Sat, 15 November 2014 01:42 Go to previous messageGo to next message
Lalit Kumar B
Messages: 3174
Registered: May 2013
Location: World Wide on the Web
Senior Member
rleishman wrote on Sat, 15 November 2014 11:25
When I saw the title, I was thinking of *real* DBA nightmares.


I thought twice before entering the topic title, but then I went along with the way the word "DBA" is vaguely understood at my workplace Wink
Re: DBA's nightmares [message #627798 is a reply to message #627720] Mon, 17 November 2014 03:35 Go to previous messageGo to next message
gazzag
Messages: 1118
Registered: November 2010
Location: Bedwas, UK
Senior Member
Lalit Kumar B wrote on Sat, 15 November 2014 07:42
I went along with the way the word "DBA" is vaguely understood at my workplace Wink

I'm afraid that this is par for the course for the DBA. If it ultimately connects to a database (doesn't everything??) get the DBA involved at the earliest opportunity.

But your post reminded me:

Q: Why is a bad DBA so bad at personal relationships?
A: Because they never commit.

Wink
Re: DBA's nightmares [message #627853 is a reply to message #627708] Mon, 17 November 2014 08:55 Go to previous messageGo to next message
joy_division
Messages: 4963
Registered: February 2005
Location: East Coast USA
Senior Member
Barbara Boehmer wrote on Fri, 14 November 2014 21:07
I have heard worse. We generally started from the most basic: Is the computer plugged in? One person had difficulty determining whether it was plugged in or not. When asked why, it was explained that it was hard to see because the lights were out, due to the power outage.


I remember reading that exact story over 10 years ago. My memory is not the best, but I do remember that one and it was someone calling Gateway tech support about their Gateway computer.

I think the response from the tech support guy was something like, "Ok, I know the problem. Please pack up the computer and send it back because you are too dumb to own a computer."

Because of that, I just assumed it was a made up story.
Re: DBA's nightmares [message #627857 is a reply to message #627853] Mon, 17 November 2014 09:10 Go to previous messageGo to next message
gazzag
Messages: 1118
Registered: November 2010
Location: Bedwas, UK
Senior Member
There is an interesting article on a similar story here.
Re: DBA's nightmares [message #627865 is a reply to message #627798] Mon, 17 November 2014 10:20 Go to previous messageGo to next message
Lalit Kumar B
Messages: 3174
Registered: May 2013
Location: World Wide on the Web
Senior Member
gazzag wrote on Mon, 17 November 2014 15:05

Q: Why is a bad DBA so bad at personal relationships?
A: Because they never commit.


LOL Smile
Re: DBA's nightmares [message #627867 is a reply to message #627853] Mon, 17 November 2014 10:58 Go to previous message
Barbara Boehmer
Messages: 9077
Registered: November 2002
Location: California, USA
Senior Member
joy_division wrote on Mon, 17 November 2014 06:55
Barbara Boehmer wrote on Fri, 14 November 2014 21:07
I have heard worse. We generally started from the most basic: Is the computer plugged in? One person had difficulty determining whether it was plugged in or not. When asked why, it was explained that it was hard to see because the lights were out, due to the power outage.


I remember reading that exact story over 10 years ago. My memory is not the best, but I do remember that one and it was someone calling Gateway tech support about their Gateway computer.

I think the response from the tech support guy was something like, "Ok, I know the problem. Please pack up the computer and send it back because you are too dumb to own a computer."

Because of that, I just assumed it was a made up story.


I believe this is actually a common problem. Users who don't understand, expect that the computers should still work during a power outage, since the telephones still work. There are also many places that have emergency generators that automatically turn on when the power goes off and power certain important things and limited lighting for safety. So, in their defense, it is perhaps not such an unreasonable expectation that the computer should still work, especially in something like a dispatch center in a law enforcement agency, such as I worked for.


Previous Topic: Wordpress Blog - SQL ouput formatting
Next Topic: NYOUG, 11 December
Goto Forum:
  


Current Time: Thu Mar 28 18:17:32 CDT 2024